Nov 03 19:26:29.390177 ip-10-0-128-162 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:26:29.847863 ip-10-0-128-162 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:26:29.847863 ip-10-0-128-162 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:26:29.847863 ip-10-0-128-162 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:26:29.847863 ip-10-0-128-162 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:26:29.851365 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851288 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.851365 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851364 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.851365 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851369 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851372 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851375 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851378 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851381 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851386 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851390 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851393 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851396 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851400 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851402 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851405 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851408 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851411 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851414 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851422 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851425 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851428 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851430 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851433 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.851452 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851435 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851438 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851441 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851443 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851447 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851450 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851454 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851457 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851460 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851463 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851466 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851468 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851471 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851473 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851476 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851478 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851481 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851483 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851485 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851488 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.851988 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851491 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851493 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851496 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851498 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851501 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851503 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851507 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851509 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851516 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851519 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851522 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851524 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851527 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851530 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851532 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851535 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851537 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851539 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851542 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.852702 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851544 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851547 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851549 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851552 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851555 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851558 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851560 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851562 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851565 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851567 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851570 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851572 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851575 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851577 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851580 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851583 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851585 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851588 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851590 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851592 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.853182 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851595 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851598 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851602 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851605 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851608 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851610 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851613 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851615 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851618 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851620 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851623 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851625 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851628 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851630 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851633 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851635 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851638 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851641 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851644 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.853649 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851646 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851649 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851651 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851654 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851656 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.851659 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.853971 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.853984 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.853988 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.853993 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.853998 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854001 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854004 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854007 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854010 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854013 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854015 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854018 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854021 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.854112 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854023 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854026 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854028 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854031 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854033 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854036 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854039 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854041 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854044 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854047 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854050 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854053 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854056 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854058 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854060 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854063 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854065 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854068 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854071 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.854571 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854074 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854077 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854079 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854082 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854085 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854087 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854090 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854092 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854095 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854097 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854100 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854102 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854105 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854108 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854110 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854113 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854115 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854118 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854121 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.855030 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854124 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854126 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854129 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854131 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854140 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854143 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854146 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854150 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854154 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854157 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854160 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854162 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854165 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854168 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854171 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854174 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854177 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854179 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854182 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.855474 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854184 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854187 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854189 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854192 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854194 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854197 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854199 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854202 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854204 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854207 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854209 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854212 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854214 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854218 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854221 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854223 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854226 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854228 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854231 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854233 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.855941 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854236 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854238 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854240 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854243 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854245 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854248 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854251 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854253 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854256 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854259 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854261 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854268 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854271 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854274 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854276 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854279 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854757 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854762 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854765 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854768 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.856410 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854771 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854774 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854777 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854780 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854782 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854785 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854788 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854791 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854793 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854796 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854799 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854801 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854804 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854807 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854809 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854812 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854814 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854817 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854820 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854823 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.856894 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854826 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854829 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854831 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854834 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854836 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854839 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854841 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854860 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854865 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854869 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854872 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854875 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854878 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854881 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854884 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854887 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854889 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854892 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854895 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854898 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.857366 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854900 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854903 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854906 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854908 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854910 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854913 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854916 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854918 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854921 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854924 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854926 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854929 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854932 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854935 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854937 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854940 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854943 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854945 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854948 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854950 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.857878 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854953 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854955 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854958 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854961 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854963 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854966 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854968 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854971 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854973 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854976 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854978 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854984 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854987 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854990 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854992 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854994 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854997 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.854999 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855004 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.858327 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855008 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855011 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855015 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855018 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855020 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855023 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855025 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855028 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855030 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855033 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855036 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855039 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855041 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855044 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855047 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855049 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855051 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855054 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855056 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855059 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.858763 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855062 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855064 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855068 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855179 2563 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855189 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855199 2563 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855204 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855208 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855212 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855216 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855220 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855223 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855226 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855231 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855234 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855237 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855240 2563 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855243 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855247 2563 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855250 2563 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855253 2563 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855257 2563 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855260 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855263 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:26:29.859271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855266 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855269 2563 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855273 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855276 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855279 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855282 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855286 2563 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855289 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855292 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855295 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855298 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855302 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855305 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855308 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855312 2563 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855316 2563 flags.go:64] FLAG: --enable-server="true" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855319 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855324 2563 flags.go:64] FLAG: --event-burst="100" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855327 2563 flags.go:64] FLAG: --event-qps="50" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855331 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855334 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855337 2563 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855346 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855349 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855353 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:26:29.859825 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855356 2563 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855359 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855362 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855365 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855368 2563 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855371 2563 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855374 2563 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855377 2563 flags.go:64] FLAG: --feature-gates="" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855381 2563 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855384 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855387 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855390 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855393 2563 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855397 2563 flags.go:64] FLAG: --help="false" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855400 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855403 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855406 2563 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855409 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855412 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855415 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855418 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855421 2563 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855425 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:26:29.860570 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855428 2563 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855431 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855434 2563 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855440 2563 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855443 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855446 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855449 2563 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855452 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855454 2563 flags.go:64] FLAG: --lock-file="" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855457 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855460 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855464 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855469 2563 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855472 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855475 2563 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855478 2563 flags.go:64] FLAG: --logging-format="text" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855482 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855486 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855489 2563 flags.go:64] FLAG: --manifest-url="" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855491 2563 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855496 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855499 2563 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855503 2563 flags.go:64] FLAG: --max-pods="110" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855506 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855509 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:26:29.861151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855512 2563 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855515 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855519 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855522 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855525 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855532 2563 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855535 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855539 2563 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855542 2563 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855545 2563 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855548 2563 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855551 2563 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855554 2563 flags.go:64] FLAG: --port="10250" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855557 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855560 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00c47fbc4173ac84e" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855563 2563 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855566 2563 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855569 2563 flags.go:64] FLAG: --register-node="true" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855573 2563 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855576 2563 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855579 2563 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855582 2563 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855585 2563 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855589 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855592 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855596 2563 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:26:29.861751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855601 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855605 2563 flags.go:64] FLAG: --runonce="false" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855611 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855616 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855620 2563 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855623 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855626 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855629 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855632 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855635 2563 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855638 2563 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855641 2563 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855644 2563 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855646 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855651 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855654 2563 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855657 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855663 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855665 2563 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855668 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855672 2563 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855675 2563 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855677 2563 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855680 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855683 2563 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:26:29.862365 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855686 2563 flags.go:64] FLAG: --v="2" Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855690 2563 flags.go:64] FLAG: --version="false" Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855694 2563 flags.go:64] FLAG: --vmodule="" Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855698 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.855701 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855858 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855865 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855869 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855872 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855874 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855877 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855880 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855882 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855885 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855888 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855890 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855893 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855896 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855899 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855901 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.862948 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855904 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855906 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855910 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855912 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855916 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855918 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855921 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855923 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855926 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855928 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855931 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855933 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855936 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855938 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855941 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855944 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855947 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855950 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855952 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.863426 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855955 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855958 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855960 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855963 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855965 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855968 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855970 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855974 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855978 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855980 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855983 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855986 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855989 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855991 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855993 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855996 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.855999 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856002 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856005 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856007 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.863924 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856010 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856012 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856015 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856017 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856020 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856022 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856025 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856028 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856030 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856033 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856035 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856038 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856042 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856046 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856048 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856051 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856053 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856056 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856058 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.864390 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856061 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856064 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856066 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856069 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856074 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856076 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856079 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856082 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856084 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856087 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856090 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856093 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856096 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856098 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856101 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856103 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856106 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856108 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856111 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856115 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.864841 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856118 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856120 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856123 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856126 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856128 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856131 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856133 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856136 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856139 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856141 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856144 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856147 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856149 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.856177 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856218 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856222 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.865314 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856225 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856229 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856232 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856235 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856237 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856240 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856243 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856246 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856249 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856251 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856254 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856257 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856260 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856262 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856265 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856269 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856271 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856274 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856276 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856279 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.865694 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856281 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856284 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856286 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856289 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856292 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856294 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856297 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856299 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856302 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856304 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856307 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856310 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856312 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856316 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856319 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856321 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856324 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856326 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856329 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856331 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.866221 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856334 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856337 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856340 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856342 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856345 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856347 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856350 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856354 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856356 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856359 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856362 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856364 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856367 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856369 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856372 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856374 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856377 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856380 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856382 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.866701 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856386 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856388 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856391 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856393 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856396 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856398 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856402 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856405 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856407 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856410 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856413 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856417 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856421 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856424 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856428 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856431 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856434 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856436 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856439 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856442 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.867154 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856445 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856448 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856451 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856453 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856455 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856458 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856460 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856463 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856466 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856468 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856471 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856473 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856476 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856478 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856481 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856483 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856487 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856491 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856496 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.867608 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856499 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856502 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856505 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856507 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856510 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.856512 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.863151 2563 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.863167 2563 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.863182 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.863187 2563 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863246 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863252 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863255 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863258 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863262 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863265 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863267 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863270 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863272 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.868060 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863275 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863277 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863280 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863283 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863286 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863289 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863291 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863294 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863296 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863299 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863302 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863304 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863307 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863309 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863312 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863314 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863317 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863319 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863322 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863324 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.868499 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863329 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863331 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863334 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863337 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863339 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863342 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863344 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863347 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863349 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863352 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863355 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863357 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863360 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863362 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863365 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863367 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863370 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863373 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863376 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.868986 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863378 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863381 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863383 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863386 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863388 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863391 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863393 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863396 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863399 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863401 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863404 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863407 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863409 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863412 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863421 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863424 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863426 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863429 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863432 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.869454 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863434 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863437 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863439 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863442 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863445 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863447 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863450 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863452 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863455 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863458 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863460 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863462 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863465 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863467 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863470 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863472 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863475 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863477 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863480 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863482 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.869906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863485 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863487 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863490 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863494 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863498 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863501 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863504 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863507 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863518 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863523 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863526 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863529 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863531 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863534 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863537 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863540 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863542 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863546 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.870373 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863549 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.863577 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863777 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863781 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863784 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863787 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863790 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863793 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863795 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863798 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863800 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863803 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863805 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863808 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863811 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.870786 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863813 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863816 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863818 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863821 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863823 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863826 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863829 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863831 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863839 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863842 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863858 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863862 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863865 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863868 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863870 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863873 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863875 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863878 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863881 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863883 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.871157 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863886 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863888 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863891 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863895 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863899 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863902 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863905 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863907 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863910 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863913 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863916 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863918 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863921 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863924 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863927 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863929 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863932 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863935 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863938 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863940 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.871619 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863943 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863951 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863954 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863957 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863959 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863962 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863965 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863967 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863970 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863974 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863977 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863980 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863988 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863991 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863993 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863996 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.863998 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864001 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864003 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.872097 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864006 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864008 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864011 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864013 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864016 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864018 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864024 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864027 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864030 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864032 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864035 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864038 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864041 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864043 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864046 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864054 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864057 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864059 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864062 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.872576 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864064 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864067 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864069 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864072 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864074 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864077 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864079 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864082 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864084 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864087 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864089 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864092 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864095 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864097 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864100 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.864124 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:26:29.873043 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864165 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864169 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864173 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864176 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864179 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864182 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864185 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864188 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864190 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864193 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864195 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864198 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864201 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864210 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864213 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864216 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864218 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864222 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864224 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864227 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:26:29.873418 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864230 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864233 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864236 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864239 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864241 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864244 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864246 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864249 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864251 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864255 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864257 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864260 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864263 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864265 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864269 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864271 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864275 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864278 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864280 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864283 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:26:29.873952 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864286 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864288 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864291 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864293 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864296 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864298 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864306 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864309 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864312 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864314 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864317 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864319 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864322 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864324 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864327 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864329 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864332 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864334 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864337 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:26:29.874417 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864339 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864342 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864346 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864348 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864351 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864353 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864356 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864358 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864361 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864364 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864367 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864370 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864372 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864375 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864377 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864380 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864383 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864385 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864388 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:26:29.874953 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864390 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864398 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864401 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864404 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864407 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864409 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864412 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864414 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864417 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864420 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864422 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864424 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864427 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864429 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864432 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864435 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864437 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864440 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864442 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864445 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:26:29.875686 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864447 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864450 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864455 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864459 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864462 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864465 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864468 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:29.864471 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.864521 2563 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.866667 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.867693 2563 server.go:1014] "Starting client certificate rotation" Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.867810 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.867840 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.870282 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:26:29.876371 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.874858 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:26:29.884907 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.884891 2563 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:26:29.888000 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.887986 2563 log.go:25] "Validated CRI v1 image API" Nov 03 19:26:29.888813 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.888784 2563 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:26:29.891719 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.891700 2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ba1ef60-4249-49c5-a883-35beef775cbe:/dev/nvme0n1p4 dc234ffa-a6e1-468f-8a94-7ea8cd497c24:/dev/nvme0n1p3] Nov 03 19:26:29.891786 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.891718 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:26:29.896834 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.896718 2563 manager.go:217] Machine: {Timestamp:2026-11-03 19:26:29.895447261 +0000 UTC m=+0.372116771 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100663 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274b1a6c8cb83012c96d50e7b5de4f SystemUUID:ec274b1a-6c8c-b830-12c9-6d50e7b5de4f BootID:44629097-52f3-4bb1-af5d-2697e1c56e91 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f0:ed:cf:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f0:ed:cf:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:2f:d4:5f:10:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:26:29.896834 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.896832 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:26:29.896963 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.896926 2563 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:26:29.900611 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.900589 2563 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:26:29.900759 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.900611 2563 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-162.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:26:29.900814 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.900761 2563 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:26:29.900814 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.900768 2563 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:26:29.900814 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.900783 2563 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:26:29.901430 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.901416 2563 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:26:29.901895 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.901883 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:26:29.903091 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.903081 2563 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:26:29.908006 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.907993 2563 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:26:29.908123 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908113 2563 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:26:29.908162 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908144 2563 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:26:29.908162 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908154 2563 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:26:29.908226 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908166 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:26:29.908226 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908175 2563 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:26:29.908226 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908183 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:26:29.908529 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.908509 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:26:29.912591 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.912562 2563 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:26:29.918994 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.918976 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:26:29.919747 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919735 2563 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:26:29.919787 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919765 2563 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:26:29.919939 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919930 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:26:29.919975 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919945 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:26:29.919975 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919952 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:26:29.919975 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919958 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:26:29.919975 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919963 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:26:29.919975 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.919969 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:26:29.920552 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.920542 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:26:29.920580 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.920553 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:26:29.920580 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.920561 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:26:29.920580 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.920567 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:26:29.920580 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.920576 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:26:29.920698 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.920585 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:26:29.922490 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.922479 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:26:29.922523 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.922496 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:26:29.926419 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.926403 2563 server.go:1263] "Started kubelet" Nov 03 19:26:29.926497 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.926480 2563 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:26:29.926565 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.926486 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:26:29.926565 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.926545 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:26:29.926787 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.926761 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-162.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:26:29.926875 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.926821 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:26:29.926931 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.926918 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-162.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:26:29.927207 ip-10-0-128-162 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:26:29.927488 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.927473 2563 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:26:29.928962 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.928948 2563 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:26:29.929744 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.929726 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nt77" Nov 03 19:26:29.932619 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.932546 2563 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:26:29.932785 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.932762 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:26:29.933223 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.933208 2563 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:26:29.933894 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.933834 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:26:29.934079 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.933290 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:29.934209 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.934195 2563 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:26:29.934261 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.934212 2563 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:26:29.934311 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.934301 2563 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:26:29.934311 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.934310 2563 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:26:29.936307 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936289 2563 factory.go:55] Registering systemd factory Nov 03 19:26:29.936374 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936310 2563 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:26:29.936543 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936531 2563 factory.go:153] Registering CRI-O factory Nov 03 19:26:29.936586 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936548 2563 factory.go:223] Registration of the crio container factory successfully Nov 03 19:26:29.936618 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936607 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:26:29.936649 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936633 2563 factory.go:103] Registering Raw factory Nov 03 19:26:29.936649 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.936647 2563 manager.go:1199] Started watching for new ooms in manager Nov 03 19:26:29.937555 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.937536 2563 manager.go:319] Starting recovery of all containers Nov 03 19:26:29.940719 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.940696 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nt77" Nov 03 19:26:29.941478 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.941456 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-162.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:26:29.942651 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.941549 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-162.us-east-2.compute.internal.18e49f8f6b1b042b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-162.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-162.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:26:29.926372395 +0000 UTC m=+0.403041906,LastTimestamp:2026-11-03 19:26:29.926372395 +0000 UTC m=+0.403041906,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-162.us-east-2.compute.internal,}" Nov 03 19:26:29.945330 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.945304 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:26:29.950177 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.950160 2563 manager.go:324] Recovery completed Nov 03 19:26:29.954136 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.954119 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:29.956103 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.956040 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:29.956103 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.956071 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:29.956103 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.956085 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:29.958310 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.958294 2563 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:26:29.958310 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.958307 2563 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:26:29.958435 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.958317 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:29.958435 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.958325 2563 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:26:29.959478 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.959464 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:29.959532 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.959488 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:29.959532 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.959500 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:29.960550 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.960537 2563 policy_none.go:50] "Start" Nov 03 19:26:29.960550 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.960552 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:26:29.960655 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.960562 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:26:29.962615 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.962603 2563 policy_none.go:44] "Start" Nov 03 19:26:29.996506 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.996481 2563 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:26:29.998918 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.998905 2563 manager.go:341] "Starting Device Plugin manager" Nov 03 19:26:29.999431 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:29.999412 2563 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:26:29.999485 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.999435 2563 server.go:87] "Starting device plugin registration server" Nov 03 19:26:29.999703 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.999692 2563 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:26:29.999734 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.999707 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:26:30.000148 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:29.999842 2563 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:26:30.000148 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.000056 2563 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:26:30.000148 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.000068 2563 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:26:30.000643 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.000626 2563 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:26:30.000741 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.000675 2563 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.100060 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.101103 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.101134 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.101145 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.101172 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.108366 2563 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.112639 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.108390 2563 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-162.us-east-2.compute.internal\": node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.121537 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.121519 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.170899 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.170877 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:26:30.172113 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.172096 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:26:30.172113 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.172117 2563 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:26:30.172298 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.172135 2563 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:26:30.172794 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.172166 2563 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 19:26:30.179275 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.179258 2563 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:26:30.222381 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.222348 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.273898 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.273873 2563 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal"] Nov 03 19:26:30.273970 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.273931 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:30.275278 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.275263 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:30.275353 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.275291 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:30.275353 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.275301 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:30.277483 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.277471 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:30.277626 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.277612 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.277663 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.277642 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:30.278211 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.278196 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:30.278211 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.278196 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:30.278312 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.278228 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:30.278312 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.278228 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:30.278312 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.278243 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:30.278312 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.278253 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:30.280425 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.280407 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.280491 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.280434 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:26:30.281121 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.281107 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:26:30.281197 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.281133 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:26:30.281197 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.281143 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:26:30.298255 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.298227 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.304555 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.304535 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.322719 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.322700 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.337655 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.337636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.337732 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.337657 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c8881dae007849f880e1281962f0bcb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"9c8881dae007849f880e1281962f0bcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.337732 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.337674 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.422815 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.422743 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.438095 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.438075 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c8881dae007849f880e1281962f0bcb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"9c8881dae007849f880e1281962f0bcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.438162 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.438073 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c8881dae007849f880e1281962f0bcb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"9c8881dae007849f880e1281962f0bcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.438162 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.438142 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.438162 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.438153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.438268 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.438168 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.438268 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.438185 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.523225 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:30.523201 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:30.601712 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.601682 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:30.608021 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:30.607893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:32.046036 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.046018 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.146573 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.146509 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.247040 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.247019 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.274123 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.274101 2563 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:26:32.290105 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.290088 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:26:32.290214 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.290194 2563 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:26:32.290254 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.290220 2563 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:26:32.347677 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.347638 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.368846 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.368818 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:21:31 +0000 UTC" deadline="2028-06-30 16:24:54.14600929 +0000 UTC" Nov 03 19:26:32.368846 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.368840 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14516h58m21.777171359s" Nov 03 19:26:32.448725 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.448670 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.549229 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.549199 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.649321 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:32.649279 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-162.us-east-2.compute.internal\" not found" Nov 03 19:26:32.676721 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:32.676664 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852f7206562983f9f71af157dcc57010.slice/crio-677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab WatchSource:0}: Error finding container 677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab: Status 404 returned error can't find the container with id 677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab Nov 03 19:26:32.676933 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:32.676916 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8881dae007849f880e1281962f0bcb.slice/crio-0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c WatchSource:0}: Error finding container 0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c: Status 404 returned error can't find the container with id 0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c Nov 03 19:26:32.681539 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.681523 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:26:32.723710 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.723661 2563 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:26:32.755694 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.755672 2563 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:32.758682 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.758669 2563 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:26:32.763738 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.763725 2563 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:26:32.765014 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.765001 2563 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:26:32.777417 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:32.777401 2563 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:26:33.332528 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.332502 2563 apiserver.go:52] "Watching apiserver" Nov 03 19:26:33.355944 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.355919 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:26:33.367581 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.367546 2563 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:26:33.377839 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.376918 2563 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:26:33.377966 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.377825 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-s9hlp","openshift-dns/node-resolver-dmtq4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-m8xhh","openshift-multus/network-metrics-daemon-jsx92","openshift-network-operator/iptables-alerter-x2xzc","openshift-ovn-kubernetes/ovnkube-node-6k5vq","kube-system/konnectivity-agent-f5dt8","kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal","openshift-multus/multus-vf8zn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk","openshift-cluster-node-tuning-operator/tuned-8x5vv","openshift-image-registry/node-ca-22xws","openshift-network-diagnostics/network-check-target-dsngn"] Nov 03 19:26:33.380912 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.380883 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.383281 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.383257 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.384190 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384170 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s8pjd\"" Nov 03 19:26:33.384286 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384180 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:26:33.384286 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384263 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:26:33.384397 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384352 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.384397 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384390 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.384496 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384412 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:26:33.384539 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.384526 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:26:33.385723 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.385701 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:33.385811 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.385767 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:33.386084 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.386065 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:26:33.386187 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.386175 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmsjd\"" Nov 03 19:26:33.386262 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.386241 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.386366 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.386303 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.388068 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.388048 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.388587 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.388566 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sctzr" Nov 03 19:26:33.390295 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.390280 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:33.390380 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.390342 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:33.392823 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.392494 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.394767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.394747 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.398808 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.398788 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sctzr" Nov 03 19:26:33.399479 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.399169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.399479 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.399235 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:33.402066 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.402047 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.405473 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.404520 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.407151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.406785 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.409275 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.409252 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.456845 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.456826 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:26:33.468102 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-kubelet\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.468212 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cnibin\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.468212 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-netns\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.468212 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-conf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.468351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468220 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-ovn\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.468351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl24g\" (UniqueName: \"kubernetes.io/projected/3fe747e4-481a-461c-9b8a-a85e9f4957d5-kube-api-access-gl24g\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.468351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.468351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468346 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-node-log\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.468481 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-log-socket\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.468481 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89460d5b-9bf2-4029-9d64-1e21733554d0-tmp-dir\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.468481 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4936d277-165f-4cf6-a9cb-6930c76e875a-agent-certs\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.468602 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.468602 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468533 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-etc-kubernetes\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.468602 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5832527a-10bb-4d88-b8e8-c473df01d219-iptables-alerter-script\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.468767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.468767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468673 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-multus-certs\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.468767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468698 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.468767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468722 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-bin\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.468880 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468775 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.468880 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.468937 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrwr\" (UniqueName: \"kubernetes.io/projected/704cd8b4-7baa-493a-9813-650106a42d13-kube-api-access-8xrwr\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.468985 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.469033 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.468996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-netd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.469065 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469034 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.469095 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469067 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-dbus\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.469126 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469103 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-host\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.469158 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469132 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-slash\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.469185 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-config\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.469216 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469183 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7m8\" (UniqueName: \"kubernetes.io/projected/5832527a-10bb-4d88-b8e8-c473df01d219-kube-api-access-7l7m8\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.469216 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469210 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmrn\" (UniqueName: \"kubernetes.io/projected/89460d5b-9bf2-4029-9d64-1e21733554d0-kube-api-access-nmmrn\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.469308 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469237 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-registration-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.469308 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6862\" (UniqueName: \"kubernetes.io/projected/abf3110c-9189-4738-ad6d-c70d1178ed91-kube-api-access-f6862\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:33.469406 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5832527a-10bb-4d88-b8e8-c473df01d219-host-slash\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.469406 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-tuned\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.469406 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469388 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkkf\" (UniqueName: \"kubernetes.io/projected/b22777af-ae5b-4635-a4ff-140ae4be22e9-kube-api-access-xfkkf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.469543 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21d46003-1ed7-40dd-b4b8-1e3a05448eee-host\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.469543 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469440 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-system-cni-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.469543 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-multus-daemon-config\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.469543 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:33.469731 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-run\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.469731 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469560 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-sys\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.469731 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnx8\" (UniqueName: \"kubernetes.io/projected/c3132370-8820-4b1e-a9b2-adcd6fe1b958-kube-api-access-zbnx8\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.469731 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469637 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-socket-dir-parent\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.469731 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-kubelet-config\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.469731 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-cni-binary-copy\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-conf-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469772 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-device-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469795 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-kubernetes\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-env-overrides\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469911 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-multus\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470037 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.469948 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-lib-modules\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-systemd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-etc-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470100 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-netns\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21d46003-1ed7-40dd-b4b8-1e3a05448eee-serviceca\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4936d277-165f-4cf6-a9cb-6930c76e875a-konnectivity-ca\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470183 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-k8s-cni-cncf-io\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-hostroot\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-var-lib-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovn-node-metrics-cert\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-script-lib\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65xr\" (UniqueName: \"kubernetes.io/projected/21d46003-1ed7-40dd-b4b8-1e3a05448eee-kube-api-access-m65xr\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-system-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-socket-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470445 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-var-lib-kubelet\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.470514 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470480 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-tmp\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-kubelet\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89460d5b-9bf2-4029-9d64-1e21733554d0-hosts-file\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470609 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-sys-fs\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470641 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-cnibin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.470776 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470750 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcklg\" (UniqueName: \"kubernetes.io/projected/9b2568c8-4309-407c-a381-83a1c8712e7d-kube-api-access-vcklg\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.471106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-modprobe-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.471106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysconfig\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.471106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-systemd-units\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.471106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.471106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-systemd\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.471106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.470919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-bin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.484757 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.484739 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hwgf7\"" Nov 03 19:26:33.484862 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.484781 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:26:33.484862 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.484816 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6kk8\"" Nov 03 19:26:33.484862 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.484855 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:26:33.484954 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.484919 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.485028 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485016 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:26:33.485063 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485047 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.485110 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485101 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.485141 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485121 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:26:33.485334 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485323 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:26:33.485367 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485350 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.485404 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485395 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:26:33.485434 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485427 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6b5k\"" Nov 03 19:26:33.485486 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485477 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.485516 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485504 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:26:33.485558 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485549 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lsdlf\"" Nov 03 19:26:33.485598 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485589 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xp5ps\"" Nov 03 19:26:33.485641 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485633 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.485680 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485672 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:26:33.485718 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485709 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.485808 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485798 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-st58b\"" Nov 03 19:26:33.485880 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485871 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8cx9\"" Nov 03 19:26:33.485911 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485896 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.485940 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485930 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:26:33.485997 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.485951 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:26:33.571316 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571285 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21d46003-1ed7-40dd-b4b8-1e3a05448eee-serviceca\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4936d277-165f-4cf6-a9cb-6930c76e875a-konnectivity-ca\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-k8s-cni-cncf-io\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-hostroot\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-var-lib-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovn-node-metrics-cert\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-script-lib\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.571482 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65xr\" (UniqueName: \"kubernetes.io/projected/21d46003-1ed7-40dd-b4b8-1e3a05448eee-kube-api-access-m65xr\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571473 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-system-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-socket-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-var-lib-kubelet\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571581 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-tmp\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-kubelet\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89460d5b-9bf2-4029-9d64-1e21733554d0-hosts-file\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-sys-fs\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-cnibin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcklg\" (UniqueName: \"kubernetes.io/projected/9b2568c8-4309-407c-a381-83a1c8712e7d-kube-api-access-vcklg\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-modprobe-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571795 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysconfig\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.571822 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21d46003-1ed7-40dd-b4b8-1e3a05448eee-serviceca\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571876 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-systemd-units\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571899 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysconfig\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571904 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-systemd\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571947 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571963 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-bin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.571999 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-systemd-units\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-kubelet\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cnibin\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-netns\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-conf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-ovn\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl24g\" (UniqueName: \"kubernetes.io/projected/3fe747e4-481a-461c-9b8a-a85e9f4957d5-kube-api-access-gl24g\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572213 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-node-log\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-log-socket\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572266 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89460d5b-9bf2-4029-9d64-1e21733554d0-tmp-dir\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.572498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572272 2563 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4936d277-165f-4cf6-a9cb-6930c76e875a-agent-certs\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-etc-kubernetes\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5832527a-10bb-4d88-b8e8-c473df01d219-iptables-alerter-script\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-multus-certs\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572442 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572468 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-bin\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572544 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrwr\" (UniqueName: \"kubernetes.io/projected/704cd8b4-7baa-493a-9813-650106a42d13-kube-api-access-8xrwr\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-netd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572627 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-k8s-cni-cncf-io\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-dbus\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.573340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-host\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-slash\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572721 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-config\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7m8\" (UniqueName: \"kubernetes.io/projected/5832527a-10bb-4d88-b8e8-c473df01d219-kube-api-access-7l7m8\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmrn\" (UniqueName: \"kubernetes.io/projected/89460d5b-9bf2-4029-9d64-1e21733554d0-kube-api-access-nmmrn\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-registration-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6862\" (UniqueName: \"kubernetes.io/projected/abf3110c-9189-4738-ad6d-c70d1178ed91-kube-api-access-f6862\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5832527a-10bb-4d88-b8e8-c473df01d219-host-slash\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-tuned\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkkf\" (UniqueName: \"kubernetes.io/projected/b22777af-ae5b-4635-a4ff-140ae4be22e9-kube-api-access-xfkkf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21d46003-1ed7-40dd-b4b8-1e3a05448eee-host\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-system-cni-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.572994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-multus-daemon-config\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-run\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-sys\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnx8\" (UniqueName: \"kubernetes.io/projected/c3132370-8820-4b1e-a9b2-adcd6fe1b958-kube-api-access-zbnx8\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.574144 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573106 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4936d277-165f-4cf6-a9cb-6930c76e875a-konnectivity-ca\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573126 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-socket-dir-parent\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-kubelet-config\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-cni-binary-copy\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-conf-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573235 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-device-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-kubernetes\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-env-overrides\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-multus\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-lib-modules\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-systemd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-etc-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573454 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-netns\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573572 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-netns\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573619 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-kubelet\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.574728 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573671 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89460d5b-9bf2-4029-9d64-1e21733554d0-hosts-file\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573721 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-sys-fs\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573841 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-system-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573864 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.573916 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-cnibin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574191 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-dbus\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574284 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574557 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-modprobe-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574608 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-systemd\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574653 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-bin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574695 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-kubelet\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574737 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cnibin\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574777 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-netns\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574888 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-conf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574927 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-ovn\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.575385 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.574950 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-script-lib\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.576027 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.575592 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.576027 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.575662 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-node-log\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.576027 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.575711 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-log-socket\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.576663 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.576630 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-socket-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.576795 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.576661 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-hostroot\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.576795 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.576740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-var-lib-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.576805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-netd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.576887 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.577242 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.577615 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-var-lib-kubelet\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.577777 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-multus\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.577836 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-systemd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.577911 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-run\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.577962 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-host\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578028 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89460d5b-9bf2-4029-9d64-1e21733554d0-tmp-dir\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578081 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-slash\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578225 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovn-node-metrics-cert\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578414 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578523 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-config\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578541 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5832527a-10bb-4d88-b8e8-c473df01d219-host-slash\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578562 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-registration-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578590 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-sys\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.576421 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-tmp\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.579076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578775 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21d46003-1ed7-40dd-b4b8-1e3a05448eee-host\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578832 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-kubelet-config\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578906 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-system-cni-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578960 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-socket-dir-parent\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578989 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.578996 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4936d277-165f-4cf6-a9cb-6930c76e875a-agent-certs\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579037 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-conf-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579113 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-multus-certs\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579111 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-device-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579141 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-etc-kubernetes\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579487 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-cni-binary-copy\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579604 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579702 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5832527a-10bb-4d88-b8e8-c473df01d219-iptables-alerter-script\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579806 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579861 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-bin\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.579908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-etc-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.580079 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-env-overrides\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.580167 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-kubernetes\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.581336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.580178 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-lib-modules\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.582165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.580223 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.582165 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.580335 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:33.582165 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.580423 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs podName:abf3110c-9189-4738-ad6d-c70d1178ed91 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:34.080393851 +0000 UTC m=+3.134939692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs") pod "network-metrics-daemon-jsx92" (UID: "abf3110c-9189-4738-ad6d-c70d1178ed91") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:33.582165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.580575 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-multus-daemon-config\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.582165 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.580682 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:33.582165 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.580740 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret podName:9c908f51-a28e-46fb-af36-ac4cb9163671 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:34.080716048 +0000 UTC m=+3.135261883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret") pod "global-pull-secret-syncer-s9hlp" (UID: "9c908f51-a28e-46fb-af36-ac4cb9163671") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:33.583061 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.582981 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-tuned\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.590635 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.590618 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl24g\" (UniqueName: \"kubernetes.io/projected/3fe747e4-481a-461c-9b8a-a85e9f4957d5-kube-api-access-gl24g\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.598846 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.597391 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:26:33.598846 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.597458 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:26:33.598846 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.597472 2563 projected.go:196] Error preparing data for projected volume kube-api-access-tgfkb for pod openshift-network-diagnostics/network-check-target-dsngn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:33.598846 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:33.597555 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb podName:11922a7d-e991-4de5-b28e-8ce492742213 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:34.097540269 +0000 UTC m=+3.152086100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tgfkb" (UniqueName: "kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb") pod "network-check-target-dsngn" (UID: "11922a7d-e991-4de5-b28e-8ce492742213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:33.600076 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.599991 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"852f7206562983f9f71af157dcc57010","Type":"ContainerStarted","Data":"677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab"} Nov 03 19:26:33.600783 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.600758 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6862\" (UniqueName: \"kubernetes.io/projected/abf3110c-9189-4738-ad6d-c70d1178ed91-kube-api-access-f6862\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:33.601291 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.601130 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmrn\" (UniqueName: \"kubernetes.io/projected/89460d5b-9bf2-4029-9d64-1e21733554d0-kube-api-access-nmmrn\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.601765 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.601623 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"9c8881dae007849f880e1281962f0bcb","Type":"ContainerStarted","Data":"0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c"} Nov 03 19:26:33.601765 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.601707 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnx8\" (UniqueName: \"kubernetes.io/projected/c3132370-8820-4b1e-a9b2-adcd6fe1b958-kube-api-access-zbnx8\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.604064 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.602221 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkkf\" (UniqueName: \"kubernetes.io/projected/b22777af-ae5b-4635-a4ff-140ae4be22e9-kube-api-access-xfkkf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.604064 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.603809 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrwr\" (UniqueName: \"kubernetes.io/projected/704cd8b4-7baa-493a-9813-650106a42d13-kube-api-access-8xrwr\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.604757 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.604712 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcklg\" (UniqueName: \"kubernetes.io/projected/9b2568c8-4309-407c-a381-83a1c8712e7d-kube-api-access-vcklg\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.606154 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.606131 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65xr\" (UniqueName: \"kubernetes.io/projected/21d46003-1ed7-40dd-b4b8-1e3a05448eee-kube-api-access-m65xr\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.606664 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.606643 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7m8\" (UniqueName: \"kubernetes.io/projected/5832527a-10bb-4d88-b8e8-c473df01d219-kube-api-access-7l7m8\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.624635 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.624611 2563 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:26:33.710566 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.710536 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22xws" Nov 03 19:26:33.715488 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.715463 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:33.721646 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:33.721618 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d46003_1ed7_40dd_b4b8_1e3a05448eee.slice/crio-d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90 WatchSource:0}: Error finding container d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90: Status 404 returned error can't find the container with id d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90 Nov 03 19:26:33.727409 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.727387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:26:33.732224 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.732192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:26:33.733987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.733950 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:33.737315 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.737294 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:26:33.741459 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:33.741433 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5832527a_10bb_4d88_b8e8_c473df01d219.slice/crio-4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6 WatchSource:0}: Error finding container 4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6: Status 404 returned error can't find the container with id 4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6 Nov 03 19:26:33.742352 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:33.742295 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3132370_8820_4b1e_a9b2_adcd6fe1b958.slice/crio-2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491 WatchSource:0}: Error finding container 2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491: Status 404 returned error can't find the container with id 2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491 Nov 03 19:26:33.746222 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.746145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vf8zn" Nov 03 19:26:33.750370 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:33.750342 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89460d5b_9bf2_4029_9d64_1e21733554d0.slice/crio-a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986 WatchSource:0}: Error finding container a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986: Status 404 returned error can't find the container with id a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986 Nov 03 19:26:33.758065 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:33.758044 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704cd8b4_7baa_493a_9813_650106a42d13.slice/crio-76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8 WatchSource:0}: Error finding container 76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8: Status 404 returned error can't find the container with id 76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8 Nov 03 19:26:33.758955 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.758668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:26:33.761512 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:33.761477 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:26:33.769586 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:26:33.769568 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2568c8_4309_407c_a381_83a1c8712e7d.slice/crio-b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867 WatchSource:0}: Error finding container b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867: Status 404 returned error can't find the container with id b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867 Nov 03 19:26:34.178435 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.178405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:34.178555 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.178452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:34.178555 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.178487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:34.178671 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.178575 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:34.178671 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.178613 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret podName:9c908f51-a28e-46fb-af36-ac4cb9163671 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:35.178601783 +0000 UTC m=+4.233147606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret") pod "global-pull-secret-syncer-s9hlp" (UID: "9c908f51-a28e-46fb-af36-ac4cb9163671") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:34.178919 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.178905 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:34.178953 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.178943 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs podName:abf3110c-9189-4738-ad6d-c70d1178ed91 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:35.178933762 +0000 UTC m=+4.233479586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs") pod "network-metrics-daemon-jsx92" (UID: "abf3110c-9189-4738-ad6d-c70d1178ed91") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:34.179030 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.179019 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:26:34.179072 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.179035 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:26:34.179072 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.179044 2563 projected.go:196] Error preparing data for projected volume kube-api-access-tgfkb for pod openshift-network-diagnostics/network-check-target-dsngn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:34.179072 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:34.179066 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb podName:11922a7d-e991-4de5-b28e-8ce492742213 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:35.17905935 +0000 UTC m=+4.233605170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tgfkb" (UniqueName: "kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb") pod "network-check-target-dsngn" (UID: "11922a7d-e991-4de5-b28e-8ce492742213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:34.402270 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.402218 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:21:33 +0000 UTC" deadline="2028-08-16 08:25:19.30203635 +0000 UTC" Nov 03 19:26:34.402270 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.402271 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15636h58m44.899769172s" Nov 03 19:26:34.616861 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.616741 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmtq4" event={"ID":"89460d5b-9bf2-4029-9d64-1e21733554d0","Type":"ContainerStarted","Data":"a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986"} Nov 03 19:26:34.619623 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.619590 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerStarted","Data":"2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491"} Nov 03 19:26:34.622151 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.622126 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22xws" event={"ID":"21d46003-1ed7-40dd-b4b8-1e3a05448eee","Type":"ContainerStarted","Data":"d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90"} Nov 03 19:26:34.629831 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.629803 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" event={"ID":"b22777af-ae5b-4635-a4ff-140ae4be22e9","Type":"ContainerStarted","Data":"24657063d6dc00cc0e5b9e8eb27b0adcbdc499111ac3b576f322d492349298ae"} Nov 03 19:26:34.632964 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.632938 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vf8zn" event={"ID":"704cd8b4-7baa-493a-9813-650106a42d13","Type":"ContainerStarted","Data":"76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8"} Nov 03 19:26:34.636766 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.635740 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f5dt8" event={"ID":"4936d277-165f-4cf6-a9cb-6930c76e875a","Type":"ContainerStarted","Data":"7f2438bb68eb6abf317a63fd27bcb696ecf96690f1e0f2898760efadd7c5c636"} Nov 03 19:26:34.644474 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.644416 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x2xzc" event={"ID":"5832527a-10bb-4d88-b8e8-c473df01d219","Type":"ContainerStarted","Data":"4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6"} Nov 03 19:26:34.655612 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.655588 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"6b635637989e310c529c823dc2ca0fc8cbbb61d9dee2ba23e2e22204149dbea0"} Nov 03 19:26:34.669530 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.668828 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"9c8881dae007849f880e1281962f0bcb","Type":"ContainerStarted","Data":"6c09dcb24bcad9a60a4d97609cd7f7b2e283f805d065b6a0b5a0d2ecba39351c"} Nov 03 19:26:34.685670 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.685141 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" podStartSLOduration=2.685125903 podStartE2EDuration="2.685125903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:34.684051475 +0000 UTC m=+3.738597318" watchObservedRunningTime="2026-11-03 19:26:34.685125903 +0000 UTC m=+3.739671746" Nov 03 19:26:34.695998 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:34.695886 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" event={"ID":"9b2568c8-4309-407c-a381-83a1c8712e7d","Type":"ContainerStarted","Data":"b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867"} Nov 03 19:26:35.190544 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.190505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:35.190718 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.190567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:35.190718 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.190621 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:35.190829 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.190751 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:35.190829 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.190811 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret podName:9c908f51-a28e-46fb-af36-ac4cb9163671 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:37.190793651 +0000 UTC m=+6.245339486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret") pod "global-pull-secret-syncer-s9hlp" (UID: "9c908f51-a28e-46fb-af36-ac4cb9163671") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:35.191141 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.191125 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:26:35.191212 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.191147 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:26:35.191212 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.191163 2563 projected.go:196] Error preparing data for projected volume kube-api-access-tgfkb for pod openshift-network-diagnostics/network-check-target-dsngn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:35.191212 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.191209 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb podName:11922a7d-e991-4de5-b28e-8ce492742213 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:37.191194885 +0000 UTC m=+6.245740721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tgfkb" (UniqueName: "kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb") pod "network-check-target-dsngn" (UID: "11922a7d-e991-4de5-b28e-8ce492742213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:35.191360 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.191272 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:35.191360 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.191304 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs podName:abf3110c-9189-4738-ad6d-c70d1178ed91 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:37.191292836 +0000 UTC m=+6.245838671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs") pod "network-metrics-daemon-jsx92" (UID: "abf3110c-9189-4738-ad6d-c70d1178ed91") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:35.393858 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.393825 2563 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:26:35.403298 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.403243 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:21:33 +0000 UTC" deadline="2028-04-05 11:11:17.723675858 +0000 UTC" Nov 03 19:26:35.403298 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.403296 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12447h44m42.320383634s" Nov 03 19:26:35.597522 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.597458 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:35.597662 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.597592 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:35.597727 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.597669 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:35.597775 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.597741 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:35.597834 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.597805 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:35.597914 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:35.597887 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:35.707357 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.707320 2563 generic.go:356] "Generic (PLEG): container finished" podID="852f7206562983f9f71af157dcc57010" containerID="d42a014f266ee361e442a8c7fb0b3a97b0f166d34e9860edaf1194a7cddec1df" exitCode=0 Nov 03 19:26:35.707525 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:35.707426 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"852f7206562983f9f71af157dcc57010","Type":"ContainerDied","Data":"d42a014f266ee361e442a8c7fb0b3a97b0f166d34e9860edaf1194a7cddec1df"} Nov 03 19:26:36.717516 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:36.717480 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"852f7206562983f9f71af157dcc57010","Type":"ContainerStarted","Data":"d76d4097a855ffb08087e9b2efcb71f1828f3479585d85e9870e6295e718d181"} Nov 03 19:26:36.721548 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:36.721519 2563 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="94fdfef6fce65119bc341c4c0d878d286de85164328dbbc86888b4f28d4ebfb8" exitCode=0 Nov 03 19:26:36.721684 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:36.721561 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerDied","Data":"94fdfef6fce65119bc341c4c0d878d286de85164328dbbc86888b4f28d4ebfb8"} Nov 03 19:26:36.778937 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:36.778852 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" podStartSLOduration=4.778826686 podStartE2EDuration="4.778826686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:36.734109542 +0000 UTC m=+5.788655384" watchObservedRunningTime="2026-11-03 19:26:36.778826686 +0000 UTC m=+5.833372531" Nov 03 19:26:37.207863 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.207786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:37.207863 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.207839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:37.208115 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.207871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:37.208115 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208051 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:26:37.208115 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208071 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:26:37.208115 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208084 2563 projected.go:196] Error preparing data for projected volume kube-api-access-tgfkb for pod openshift-network-diagnostics/network-check-target-dsngn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:37.208294 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208141 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb podName:11922a7d-e991-4de5-b28e-8ce492742213 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:41.208122164 +0000 UTC m=+10.262668001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tgfkb" (UniqueName: "kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb") pod "network-check-target-dsngn" (UID: "11922a7d-e991-4de5-b28e-8ce492742213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:37.208620 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208588 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:37.208727 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208639 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret podName:9c908f51-a28e-46fb-af36-ac4cb9163671 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:41.208625539 +0000 UTC m=+10.263171360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret") pod "global-pull-secret-syncer-s9hlp" (UID: "9c908f51-a28e-46fb-af36-ac4cb9163671") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:37.208727 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208704 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:37.208835 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.208741 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs podName:abf3110c-9189-4738-ad6d-c70d1178ed91 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:41.208729106 +0000 UTC m=+10.263274926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs") pod "network-metrics-daemon-jsx92" (UID: "abf3110c-9189-4738-ad6d-c70d1178ed91") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:37.600481 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.598942 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:37.600481 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.599099 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:37.600481 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.599487 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:37.600481 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.599582 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:37.600481 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.599636 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:37.600481 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:37.599697 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:37.733492 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:37.733332 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f5dt8" event={"ID":"4936d277-165f-4cf6-a9cb-6930c76e875a","Type":"ContainerStarted","Data":"6ae4faada40fb4614c21f23965d69bcbd9adce72657fed6f185855a9e87c4f5c"} Nov 03 19:26:39.596740 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.596712 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:39.597200 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:39.596838 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:39.597262 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.597204 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:39.597520 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:39.597301 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:39.597520 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.597358 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:39.597520 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:39.597437 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:39.739746 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.739610 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" event={"ID":"9b2568c8-4309-407c-a381-83a1c8712e7d","Type":"ContainerStarted","Data":"1f84b95cf867c0afc9f020a93783f8919c0df6ca86cee90db8c9b1efc4a98d55"} Nov 03 19:26:39.742105 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.741856 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22xws" event={"ID":"21d46003-1ed7-40dd-b4b8-1e3a05448eee","Type":"ContainerStarted","Data":"1cd44ab9d7f65055706a05bef0586815e4c75741557571637ea404577c5de8da"} Nov 03 19:26:39.757624 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.757181 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f5dt8" podStartSLOduration=5.093035243 podStartE2EDuration="8.757169024s" totalImagesPullingTime="3.664133781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:37.749917341 +0000 UTC m=+6.804463183" watchObservedRunningTime="2026-11-03 19:26:39.757169024 +0000 UTC m=+8.811714866" Nov 03 19:26:39.757624 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:39.757469 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-22xws" podStartSLOduration=3.801779728 podStartE2EDuration="8.757462325s" totalImagesPullingTime="4.955682597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:39.756854043 +0000 UTC m=+8.811399887" watchObservedRunningTime="2026-11-03 19:26:39.757462325 +0000 UTC m=+8.812008195" Nov 03 19:26:40.011950 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:40.011898 2563 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:40.012785 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:40.012766 2563 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:41.246067 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.246039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:41.246508 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.246095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:41.246508 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.246135 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:41.246508 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246288 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:26:41.246508 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246308 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:26:41.246508 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246320 2563 projected.go:196] Error preparing data for projected volume kube-api-access-tgfkb for pod openshift-network-diagnostics/network-check-target-dsngn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:41.246508 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246375 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb podName:11922a7d-e991-4de5-b28e-8ce492742213 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:49.246357618 +0000 UTC m=+18.300903457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tgfkb" (UniqueName: "kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb") pod "network-check-target-dsngn" (UID: "11922a7d-e991-4de5-b28e-8ce492742213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:41.246824 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246739 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:41.246824 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246780 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret podName:9c908f51-a28e-46fb-af36-ac4cb9163671 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:49.24676719 +0000 UTC m=+18.301313024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret") pod "global-pull-secret-syncer-s9hlp" (UID: "9c908f51-a28e-46fb-af36-ac4cb9163671") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:41.246941 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246831 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:41.246941 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.246863 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs podName:abf3110c-9189-4738-ad6d-c70d1178ed91 nodeName:}" failed. No retries permitted until 2026-11-03 19:26:49.246855835 +0000 UTC m=+18.301401667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs") pod "network-metrics-daemon-jsx92" (UID: "abf3110c-9189-4738-ad6d-c70d1178ed91") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:41.380520 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.380442 2563 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:26:41.427416 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.427342 2563 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:26:41.380456572Z","UUID":"5c0f2a1a-e72a-4cfa-ad5f-00d200dc5573","Handler":null,"Name":"","Endpoint":""} Nov 03 19:26:41.429668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.429194 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:26:41.429668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.429215 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:26:41.615729 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.615698 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:41.615893 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.615825 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:41.617278 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.616470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:41.617278 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.616601 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:41.617278 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.616672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:41.617278 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:41.616795 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:41.747772 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.747742 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vf8zn" event={"ID":"704cd8b4-7baa-493a-9813-650106a42d13","Type":"ContainerStarted","Data":"be450e89bab722b1739f7cd824cb3fd667488e9932a49c27a5489fdcc29012fc"} Nov 03 19:26:41.751407 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.751382 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" event={"ID":"9b2568c8-4309-407c-a381-83a1c8712e7d","Type":"ContainerStarted","Data":"e2063e708cc6940052d4998949e1a4f26d2b4fade5122054f3e480ce7c3af543"} Nov 03 19:26:41.760307 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.759914 2563 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="63d0796e4b04aef9e9f53c84cfe4ff687bcde3b4392b59993677216edc8c258e" exitCode=0 Nov 03 19:26:41.760307 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.759942 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerDied","Data":"63d0796e4b04aef9e9f53c84cfe4ff687bcde3b4392b59993677216edc8c258e"} Nov 03 19:26:41.766492 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.766441 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vf8zn" podStartSLOduration=3.0978272589999998 podStartE2EDuration="10.766424926s" totalImagesPullingTime="7.668597667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:41.765899073 +0000 UTC m=+10.820444926" watchObservedRunningTime="2026-11-03 19:26:41.766424926 +0000 UTC m=+10.820970770" Nov 03 19:26:41.802273 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:41.802220 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dmtq4" podStartSLOduration=3.015505981 podStartE2EDuration="10.802203041s" totalImagesPullingTime="7.78669706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:41.802056609 +0000 UTC m=+10.856602461" watchObservedRunningTime="2026-11-03 19:26:41.802203041 +0000 UTC m=+10.856748886" Nov 03 19:26:42.765098 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:42.764882 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" event={"ID":"9b2568c8-4309-407c-a381-83a1c8712e7d","Type":"ContainerStarted","Data":"c0077eecabdd3d0a9d916bd4220836ff2f96a22dbd3bfaf298d73b220f6af2ad"} Nov 03 19:26:42.767778 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:42.767749 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmtq4" event={"ID":"89460d5b-9bf2-4029-9d64-1e21733554d0","Type":"ContainerStarted","Data":"deab4445f1c3ab780ea231e0da087140f94654b2c36ae54c6e3d417fd6ac1670"} Nov 03 19:26:42.771191 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:42.771167 2563 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="459284b853a35aca369122c330b45d8288d8be3b6e5c8dc0a2bb835742b31003" exitCode=0 Nov 03 19:26:42.771283 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:42.771252 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerDied","Data":"459284b853a35aca369122c330b45d8288d8be3b6e5c8dc0a2bb835742b31003"} Nov 03 19:26:42.784694 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:42.784656 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" podStartSLOduration=3.651126732 podStartE2EDuration="11.784644046s" totalImagesPullingTime="8.133517314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:42.78424176 +0000 UTC m=+11.838787599" watchObservedRunningTime="2026-11-03 19:26:42.784644046 +0000 UTC m=+11.839189898" Nov 03 19:26:43.595323 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:43.595249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:43.595498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:43.595333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:43.595498 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:43.595343 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:43.595498 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:43.595465 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:43.595625 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:43.595508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:43.595625 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:43.595588 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:43.776397 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:43.776336 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" event={"ID":"b22777af-ae5b-4635-a4ff-140ae4be22e9","Type":"ContainerStarted","Data":"e5fd6873f05dbb303980dcfd9cf8499ffaa3c7742faa5d3af6f90fa476090db1"} Nov 03 19:26:43.779102 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:43.779043 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x2xzc" event={"ID":"5832527a-10bb-4d88-b8e8-c473df01d219","Type":"ContainerStarted","Data":"8f29c56b8ef833908bfe680180ff75edfd31f5cd66de201a42770c369e2dccb0"} Nov 03 19:26:43.794768 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:43.794727 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x2xzc" podStartSLOduration=4.782038882 podStartE2EDuration="12.794714755s" totalImagesPullingTime="8.012675873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:43.794484004 +0000 UTC m=+12.849029847" watchObservedRunningTime="2026-11-03 19:26:43.794714755 +0000 UTC m=+12.849260596" Nov 03 19:26:44.783446 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:44.783414 2563 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="e2b02e35f1fe1c1bd7d047f3e89a5a40da5ee13778bc1d524108b680d97747c1" exitCode=0 Nov 03 19:26:44.784000 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:44.783462 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerDied","Data":"e2b02e35f1fe1c1bd7d047f3e89a5a40da5ee13778bc1d524108b680d97747c1"} Nov 03 19:26:44.806148 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:44.806100 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" podStartSLOduration=3.8942973309999998 podStartE2EDuration="13.80608326s" totalImagesPullingTime="9.911785929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:44.805263165 +0000 UTC m=+13.859809007" watchObservedRunningTime="2026-11-03 19:26:44.80608326 +0000 UTC m=+13.860629102" Nov 03 19:26:45.595387 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:45.595354 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:45.595564 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:45.595354 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:45.595564 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:45.595495 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:45.595718 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:45.595578 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:45.595718 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:45.595361 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:45.595718 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:45.595675 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:46.798848 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:46.798595 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerStarted","Data":"2862140de656b528eef3f1e4b55765f9e54853425991825c943d7915c9b84d15"} Nov 03 19:26:47.594715 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:47.594687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:47.594857 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:47.594738 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:47.594857 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:47.594746 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:47.594857 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:47.594833 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:47.594959 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:47.594901 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:47.595035 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:47.595009 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:47.803342 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:47.803313 2563 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="2862140de656b528eef3f1e4b55765f9e54853425991825c943d7915c9b84d15" exitCode=0 Nov 03 19:26:47.803695 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:47.803353 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerDied","Data":"2862140de656b528eef3f1e4b55765f9e54853425991825c943d7915c9b84d15"} Nov 03 19:26:48.807892 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.807814 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"ddf52dbc86d2914d5d512e4278984b5b87a6cbcc91817a1a926fe05883fbab54"} Nov 03 19:26:48.808319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.807931 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"3f6194379db783ea57f085ab9d8d5dd4f12f466742e15ef55bd3d740b0386dd0"} Nov 03 19:26:48.808319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.807966 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"b9d0da48dec3f756ceddc4fd87c41968e0a2e2d8edbf39604801b75f24da8d28"} Nov 03 19:26:48.808319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.808038 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"f7ece8441a2735b0329be2e97703645896b5388032d9b4c968ee7246aa7afc47"} Nov 03 19:26:48.808319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.808072 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"ba32b30c764653f71e692aaeea9fa197e27b68640ec299f27204b111b26b18e2"} Nov 03 19:26:48.808319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.808102 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"c86dfff37ec36039ed5c2e5e4d20e2e218021ac287a5d41a0300c237060ebb6a"} Nov 03 19:26:48.810844 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.810811 2563 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="f59aa3333c649833f3c6b6e9999ecfcc37fac80ce04d795558b41cc59d88a209" exitCode=0 Nov 03 19:26:48.810946 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:48.810859 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerDied","Data":"f59aa3333c649833f3c6b6e9999ecfcc37fac80ce04d795558b41cc59d88a209"} Nov 03 19:26:49.307646 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.307568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:49.307646 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.307621 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:49.307646 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.307643 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:49.307831 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307724 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:49.307831 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307771 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs podName:abf3110c-9189-4738-ad6d-c70d1178ed91 nodeName:}" failed. No retries permitted until 2026-11-03 19:27:05.307758284 +0000 UTC m=+34.362304104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs") pod "network-metrics-daemon-jsx92" (UID: "abf3110c-9189-4738-ad6d-c70d1178ed91") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:26:49.307831 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307723 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:26:49.307831 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307825 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:26:49.307948 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307838 2563 projected.go:196] Error preparing data for projected volume kube-api-access-tgfkb for pod openshift-network-diagnostics/network-check-target-dsngn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:49.307948 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307723 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:49.307948 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307879 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb podName:11922a7d-e991-4de5-b28e-8ce492742213 nodeName:}" failed. No retries permitted until 2026-11-03 19:27:05.30786529 +0000 UTC m=+34.362411134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tgfkb" (UniqueName: "kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb") pod "network-check-target-dsngn" (UID: "11922a7d-e991-4de5-b28e-8ce492742213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:26:49.307948 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.307928 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret podName:9c908f51-a28e-46fb-af36-ac4cb9163671 nodeName:}" failed. No retries permitted until 2026-11-03 19:27:05.307916274 +0000 UTC m=+34.362462094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret") pod "global-pull-secret-syncer-s9hlp" (UID: "9c908f51-a28e-46fb-af36-ac4cb9163671") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:26:49.595470 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.595386 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:49.595594 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.595527 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:49.595594 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.595535 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:49.595681 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.595533 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:49.595681 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.595609 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:49.595681 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:49.595673 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:49.816159 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.816122 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" event={"ID":"c3132370-8820-4b1e-a9b2-adcd6fe1b958","Type":"ContainerStarted","Data":"4129cb7974ecd96a6496a82d79f1f098339a4cc04214650d0f74875100b3185e"} Nov 03 19:26:49.839156 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:49.839108 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m8xhh" podStartSLOduration=7.991907589 podStartE2EDuration="18.839094969s" totalImagesPullingTime="10.84718738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:49.838605502 +0000 UTC m=+18.893151344" watchObservedRunningTime="2026-11-03 19:26:49.839094969 +0000 UTC m=+18.893640811" Nov 03 19:26:50.821316 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:50.821231 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"1fc2c7427c2f7f0f295f073c8a5d8f862f81d1fcc43ed2dd032caaceebbd7272"} Nov 03 19:26:51.595871 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:51.595830 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:51.596090 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:51.595935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:51.596090 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:51.595938 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:51.596090 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:51.595966 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:51.596090 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:51.596045 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:51.596341 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:51.596124 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:52.828614 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:52.828525 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" event={"ID":"3fe747e4-481a-461c-9b8a-a85e9f4957d5","Type":"ContainerStarted","Data":"cd79a16dcfb99153c14cb31129ebdf97c9861e1f0c45af8ff1d264e3d58d4aa1"} Nov 03 19:26:52.829358 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:52.828800 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:52.829358 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:52.828866 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:52.844588 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:52.844435 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:52.860305 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:52.860268 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" podStartSLOduration=7.499118518 podStartE2EDuration="21.860258143s" totalImagesPullingTime="14.361139625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:52.859329703 +0000 UTC m=+21.913875545" watchObservedRunningTime="2026-11-03 19:26:52.860258143 +0000 UTC m=+21.914803985" Nov 03 19:26:53.594615 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:53.594587 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:53.594775 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:53.594632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:53.594775 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:53.594737 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:53.594867 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:53.594811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:53.594867 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:53.594805 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:53.594935 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:53.594890 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:53.831163 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:53.831132 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:53.845642 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:53.845581 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:26:54.475275 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:54.475245 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:54.475436 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:54.475379 2563 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:26:54.475758 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:54.475736 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:26:55.594788 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:55.594753 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:55.594788 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:55.594788 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:55.595319 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:55.594877 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:55.595319 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:55.594939 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:55.595319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:55.595015 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:55.595319 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:55.595097 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:57.327347 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:57.327316 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dsngn"] Nov 03 19:26:57.327768 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:57.327491 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:57.327768 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:57.327601 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:57.331332 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:57.331301 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s9hlp"] Nov 03 19:26:57.331453 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:57.331429 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:57.331558 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:57.331538 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:57.332190 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:57.332166 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jsx92"] Nov 03 19:26:57.332302 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:57.332288 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:57.332418 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:57.332396 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:26:58.594479 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:58.594407 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:26:58.594964 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:58.594503 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:26:58.594964 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:26:58.594516 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:26:58.594964 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:58.594505 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:26:58.594964 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:58.594614 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:26:58.594964 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:26:58.594672 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:27:00.595234 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:00.595201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:00.595234 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:00.595243 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:00.595636 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:00.595299 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:00.595636 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:27:00.595311 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9hlp" podUID="9c908f51-a28e-46fb-af36-ac4cb9163671" Nov 03 19:27:00.595636 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:27:00.595375 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsx92" podUID="abf3110c-9189-4738-ad6d-c70d1178ed91" Nov 03 19:27:00.595636 ip-10-0-128-162 kubenswrapper[2563]: E1103 19:27:00.595420 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dsngn" podUID="11922a7d-e991-4de5-b28e-8ce492742213" Nov 03 19:27:02.255559 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.255494 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeReady" Nov 03 19:27:02.255955 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.255622 2563 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:27:02.310712 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.310691 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7fzhd"] Nov 03 19:27:02.368987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.368947 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xtdsf"] Nov 03 19:27:02.369147 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.369077 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.372743 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.372719 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:27:02.373088 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.373068 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6b2s9\"" Nov 03 19:27:02.373171 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.373153 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:27:02.385721 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.385703 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7fzhd"] Nov 03 19:27:02.385815 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.385735 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xtdsf"] Nov 03 19:27:02.385815 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.385754 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-npmtd"] Nov 03 19:27:02.385815 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.385802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.402351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.402333 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-npmtd"] Nov 03 19:27:02.402463 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.402425 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.473106 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473082 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:27:02.473238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473132 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:27:02.473238 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473148 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:27:02.473423 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473408 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:27:02.473468 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473434 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:27:02.473659 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473639 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gtgvx\"" Nov 03 19:27:02.473659 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473659 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:27:02.473807 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473677 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qhgjf\"" Nov 03 19:27:02.473807 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.473711 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:27:02.501102 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-config-volume\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.501203 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-cert\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.501203 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501187 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.501265 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqf89\" (UniqueName: \"kubernetes.io/projected/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-kube-api-access-wqf89\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.501265 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-metrics-tls\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.501265 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501256 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.501351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4bb\" (UniqueName: \"kubernetes.io/projected/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-kube-api-access-nt4bb\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.501351 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501316 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85lx9\" (UniqueName: \"kubernetes.io/projected/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-api-access-85lx9\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.501408 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-tmp-dir\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.501491 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501474 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-data-volume\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.501529 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.501501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-crio-socket\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.595434 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.595373 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:02.595434 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.595396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:02.595567 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.595543 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:02.601617 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.601594 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:27:02.601791 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.601770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-cert\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.601891 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.601824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.601891 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.601852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqf89\" (UniqueName: \"kubernetes.io/projected/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-kube-api-access-wqf89\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.602267 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.601896 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-metrics-tls\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.602267 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.602267 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4bb\" (UniqueName: \"kubernetes.io/projected/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-kube-api-access-nt4bb\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.602267 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85lx9\" (UniqueName: \"kubernetes.io/projected/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-api-access-85lx9\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.602267 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-tmp-dir\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.602267 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-data-volume\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.602556 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-crio-socket\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.602556 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602312 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-config-volume\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.602719 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602695 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.602767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602733 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-tmp-dir\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.602898 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602878 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-config-volume\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.603012 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.602960 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-data-volume\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.603153 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.603069 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-crio-socket\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.605094 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605076 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv5d7\"" Nov 03 19:27:02.605193 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605081 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xlp4b\"" Nov 03 19:27:02.605258 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605243 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:27:02.605306 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605247 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:27:02.605646 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605622 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.605767 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605633 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-metrics-tls\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.605826 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.605780 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-cert\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.613599 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.613573 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85lx9\" (UniqueName: \"kubernetes.io/projected/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-api-access-85lx9\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:02.614343 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.614323 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqf89\" (UniqueName: \"kubernetes.io/projected/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-kube-api-access-wqf89\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.614733 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.614708 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4bb\" (UniqueName: \"kubernetes.io/projected/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-kube-api-access-nt4bb\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.701650 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.701624 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5vlcb\"" Nov 03 19:27:02.701773 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.701742 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:27:02.702790 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.702776 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:02.711931 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.711917 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:02.760959 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:02.760927 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:03.047122 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.047057 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xtdsf"] Nov 03 19:27:03.049090 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.049044 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-npmtd"] Nov 03 19:27:03.050906 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:03.050817 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652e8df9_bec9_4bdc_9535_4f2ea47f45d6.slice/crio-fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902 WatchSource:0}: Error finding container fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902: Status 404 returned error can't find the container with id fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902 Nov 03 19:27:03.052122 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.052077 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7fzhd"] Nov 03 19:27:03.052957 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:03.052871 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9c8905_cc87_498c_8a3a_ec294b27fb9d.slice/crio-2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179 WatchSource:0}: Error finding container 2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179: Status 404 returned error can't find the container with id 2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179 Nov 03 19:27:03.056183 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:03.056098 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f99516_4d86_4bdd_8df3_39c1ec90f28a.slice/crio-f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5 WatchSource:0}: Error finding container f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5: Status 404 returned error can't find the container with id f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5 Nov 03 19:27:03.860634 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.860590 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xtdsf" event={"ID":"652e8df9-bec9-4bdc-9535-4f2ea47f45d6","Type":"ContainerStarted","Data":"fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902"} Nov 03 19:27:03.863314 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.863287 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npmtd" event={"ID":"1e9c8905-cc87-498c-8a3a-ec294b27fb9d","Type":"ContainerStarted","Data":"ccc476563e0118882eade38f362c756365f1e21c021bd3e6b20843fdc81ed609"} Nov 03 19:27:03.863439 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.863347 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npmtd" event={"ID":"1e9c8905-cc87-498c-8a3a-ec294b27fb9d","Type":"ContainerStarted","Data":"2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179"} Nov 03 19:27:03.864585 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:03.864558 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fzhd" event={"ID":"d1f99516-4d86-4bdd-8df3-39c1ec90f28a","Type":"ContainerStarted","Data":"f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5"} Nov 03 19:27:04.781516 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.781433 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5"] Nov 03 19:27:04.821933 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.821900 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5"] Nov 03 19:27:04.823872 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.823843 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:04.823872 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.823854 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm"] Nov 03 19:27:04.839757 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.839726 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:27:04.843320 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.843291 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5zzs7\"" Nov 03 19:27:04.844319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.843694 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:27:04.844319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.843754 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:27:04.844319 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.843888 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:27:04.844579 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.844559 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 19:27:04.854536 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.854512 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm"] Nov 03 19:27:04.854643 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.854563 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gdzp5"] Nov 03 19:27:04.856271 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.855772 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.873023 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.872733 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:04.882177 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.882152 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npmtd" event={"ID":"1e9c8905-cc87-498c-8a3a-ec294b27fb9d","Type":"ContainerStarted","Data":"1a495bed672f46827cff08e8c6e161bdc3490a0caa6a47a095f811bb10dfa482"} Nov 03 19:27:04.919442 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919406 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.919577 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919453 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.919577 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:04.919577 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.919577 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919563 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfcg\" (UniqueName: \"kubernetes.io/projected/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-api-access-5mfcg\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.919751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:04.919751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0133e827-7681-4c7a-a5a8-34b9c5da874c-kube-api-access-dsfch\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:04.919751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.919751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919667 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:04.919751 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.919704 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0133e827-7681-4c7a-a5a8-34b9c5da874c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:04.921264 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.921243 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:27:04.921702 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.921666 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:27:04.921797 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.921703 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pv4cw\"" Nov 03 19:27:04.921797 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.921752 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lz2t\"" Nov 03 19:27:04.921873 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.921848 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:27:04.921950 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.921930 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 19:27:04.938631 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.938600 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:27:04.938736 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:04.938704 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 19:27:05.020441 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-root\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.020606 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-textfile\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.020606 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.020724 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.020724 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-tls\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.020908 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.020908 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020776 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0133e827-7681-4c7a-a5a8-34b9c5da874c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.020908 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020820 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-sys\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.020908 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.020908 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0133e827-7681-4c7a-a5a8-34b9c5da874c-kube-api-access-dsfch\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5r5f\" (UniqueName: \"kubernetes.io/projected/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-kube-api-access-d5r5f\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.020955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-wtmp\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021087 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.021125 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfcg\" (UniqueName: \"kubernetes.io/projected/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-api-access-5mfcg\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.021414 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.021414 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021177 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-metrics-client-ca\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.021414 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021359 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.021753 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.021687 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.022670 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.022513 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0133e827-7681-4c7a-a5a8-34b9c5da874c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.023023 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.022999 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.024654 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.024631 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.024842 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.024814 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.025295 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.025273 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.026181 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.026159 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.036748 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.036679 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0133e827-7681-4c7a-a5a8-34b9c5da874c-kube-api-access-dsfch\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.040389 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.040364 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfcg\" (UniqueName: \"kubernetes.io/projected/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-api-access-5mfcg\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.121894 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.121860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-root\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122041 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.121923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-textfile\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122041 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.121966 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-tls\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122041 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-sys\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122155 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5r5f\" (UniqueName: \"kubernetes.io/projected/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-kube-api-access-d5r5f\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122155 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122155 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-wtmp\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122246 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122246 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-metrics-client-ca\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122856 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122831 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-metrics-client-ca\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.122990 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.122910 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-root\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.123196 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.123175 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-textfile\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.123900 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.123713 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-sys\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.123900 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.123717 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-wtmp\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.124140 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.124115 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.125873 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.125852 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-tls\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.125955 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.125886 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.135411 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.135388 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5r5f\" (UniqueName: \"kubernetes.io/projected/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-kube-api-access-d5r5f\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.182983 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.182945 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:05.216249 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.216223 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:05.231357 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.231331 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:05.324631 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.324555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:05.324631 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.324612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:05.324823 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.324645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:05.327508 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.327484 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:05.327626 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.327579 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:05.327720 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.327704 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:05.350623 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.350601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:05.363770 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.363751 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:05.378618 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.378595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:05.442320 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:05.442282 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87c6d25_883d_4f0d_b4d3_8c77abbb41ea.slice/crio-974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5 WatchSource:0}: Error finding container 974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5: Status 404 returned error can't find the container with id 974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5 Nov 03 19:27:05.483892 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.480103 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5"] Nov 03 19:27:05.494580 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:05.494548 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0133e827_7681_4c7a_a5a8_34b9c5da874c.slice/crio-5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722 WatchSource:0}: Error finding container 5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722: Status 404 returned error can't find the container with id 5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722 Nov 03 19:27:05.734897 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.734825 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jsx92"] Nov 03 19:27:05.737433 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.737409 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm"] Nov 03 19:27:05.739923 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:05.739878 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf3110c_9189_4738_ad6d_c70d1178ed91.slice/crio-c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4 WatchSource:0}: Error finding container c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4: Status 404 returned error can't find the container with id c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4 Nov 03 19:27:05.743391 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:05.743362 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3dcb7c_3e19_41bb_ba36_8c2afbde6bd5.slice/crio-05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e WatchSource:0}: Error finding container 05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e: Status 404 returned error can't find the container with id 05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e Nov 03 19:27:05.782558 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.782535 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s9hlp"] Nov 03 19:27:05.792010 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.791570 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dsngn"] Nov 03 19:27:05.886348 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.886267 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dsngn" event={"ID":"11922a7d-e991-4de5-b28e-8ce492742213","Type":"ContainerStarted","Data":"d1a394006db51797e72fde9e3c74f679ef276add1d254062abec54e5646d4bd9"} Nov 03 19:27:05.887625 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.887593 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" event={"ID":"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5","Type":"ContainerStarted","Data":"05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e"} Nov 03 19:27:05.889830 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.889793 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" event={"ID":"0133e827-7681-4c7a-a5a8-34b9c5da874c","Type":"ContainerStarted","Data":"61bb5d97ab1bb88651203f4f2003e009c669d8686c12d0d3f2c8b45d9b057b76"} Nov 03 19:27:05.889938 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.889849 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" event={"ID":"0133e827-7681-4c7a-a5a8-34b9c5da874c","Type":"ContainerStarted","Data":"b424540c62c86d6525f18d5545be072de15237660e8d719bf09c0a1ee4607a03"} Nov 03 19:27:05.889938 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.889878 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" event={"ID":"0133e827-7681-4c7a-a5a8-34b9c5da874c","Type":"ContainerStarted","Data":"5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722"} Nov 03 19:27:05.892472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.891941 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fzhd" event={"ID":"d1f99516-4d86-4bdd-8df3-39c1ec90f28a","Type":"ContainerStarted","Data":"bb04a2671cbeb6347f6281ca295bc72b3eb0bac1aaa61f513e542c3c64be1d9b"} Nov 03 19:27:05.892472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.892015 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fzhd" event={"ID":"d1f99516-4d86-4bdd-8df3-39c1ec90f28a","Type":"ContainerStarted","Data":"3a491b386798b3c49c2007295f0c046738088bd46b803f32bd6d4d0aeb1e2180"} Nov 03 19:27:05.892472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.892108 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:05.893402 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.893343 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s9hlp" event={"ID":"9c908f51-a28e-46fb-af36-ac4cb9163671","Type":"ContainerStarted","Data":"9c3dfdc1a4cd15b8c29e0a48ec030a7dbda7c98c9f01927dc508e37c13f5cd6d"} Nov 03 19:27:05.894946 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.894916 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsx92" event={"ID":"abf3110c-9189-4738-ad6d-c70d1178ed91","Type":"ContainerStarted","Data":"c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4"} Nov 03 19:27:05.896768 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.896737 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzp5" event={"ID":"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea","Type":"ContainerStarted","Data":"974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5"} Nov 03 19:27:05.910974 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:05.910929 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7fzhd" podStartSLOduration=1.5256022200000001 podStartE2EDuration="3.910912965s" totalImagesPullingTime="2.385310745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:05.909414737 +0000 UTC m=+34.963960582" watchObservedRunningTime="2026-11-03 19:27:05.910912965 +0000 UTC m=+34.965458811" Nov 03 19:27:06.905938 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.905862 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npmtd" event={"ID":"1e9c8905-cc87-498c-8a3a-ec294b27fb9d","Type":"ContainerStarted","Data":"764dd4217b138ef40c2aba54257a0fda9fbce60a35d384486629decf6323bf94"} Nov 03 19:27:06.912017 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.911958 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f68bfd554-gj6f6"] Nov 03 19:27:06.935509 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.935464 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f68bfd554-gj6f6"] Nov 03 19:27:06.935643 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.935617 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.947829 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.947944 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.948029 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5w6kx\"" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.948073 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.948139 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5m7m493jtqc6c\"" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.948184 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 19:27:06.948865 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.948233 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 19:27:06.967007 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:06.966943 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-npmtd" podStartSLOduration=1.634291632 podStartE2EDuration="4.966925862s" totalImagesPullingTime="3.33263423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:27:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:06.933814509 +0000 UTC m=+35.988360353" watchObservedRunningTime="2026-11-03 19:27:06.966925862 +0000 UTC m=+36.021471708" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-metrics-client-ca\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-tmp\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040477 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040619 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmhz\" (UniqueName: \"kubernetes.io/projected/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-kube-api-access-ssmhz\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.042472 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.040645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-grpc-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142116 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-metrics-client-ca\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142269 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142269 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142269 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142191 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-tmp\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142269 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142220 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142269 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142448 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142304 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142448 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142336 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmhz\" (UniqueName: \"kubernetes.io/projected/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-kube-api-access-ssmhz\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142448 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142362 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-grpc-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.142915 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.142864 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-metrics-client-ca\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.143078 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.143053 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-tmp\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.145298 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.145259 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.145717 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.145662 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.146594 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.146576 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-grpc-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.148442 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.148397 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.148529 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.148434 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.149509 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.149487 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.155338 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.155317 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmhz\" (UniqueName: \"kubernetes.io/projected/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-kube-api-access-ssmhz\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.345513 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.345234 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:07.715411 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.715373 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f68bfd554-gj6f6"] Nov 03 19:27:07.721685 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:07.721649 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92b0d62_15bf_4769_8a0c_fa53b1079ca2.slice/crio-edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f WatchSource:0}: Error finding container edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f: Status 404 returned error can't find the container with id edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f Nov 03 19:27:07.912272 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.911490 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xtdsf" event={"ID":"652e8df9-bec9-4bdc-9535-4f2ea47f45d6","Type":"ContainerStarted","Data":"78ce95750a509f870a1e7b82c624e06ac42f2ec71a2716c426c184c251ff3b3b"} Nov 03 19:27:07.914554 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.914520 2563 generic.go:356] "Generic (PLEG): container finished" podID="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" containerID="3feabb6ded2c6f35b21ca700d3b89f23bd925303c16e7836a2fad7c80d907558" exitCode=0 Nov 03 19:27:07.914651 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.914603 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzp5" event={"ID":"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea","Type":"ContainerDied","Data":"3feabb6ded2c6f35b21ca700d3b89f23bd925303c16e7836a2fad7c80d907558"} Nov 03 19:27:07.917366 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.917338 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" event={"ID":"a92b0d62-15bf-4769-8a0c-fa53b1079ca2","Type":"ContainerStarted","Data":"edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f"} Nov 03 19:27:07.949517 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:07.949464 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xtdsf" podStartSLOduration=1.590247733 podStartE2EDuration="5.949450679s" totalImagesPullingTime="4.359202946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:07.929438631 +0000 UTC m=+36.983984475" watchObservedRunningTime="2026-11-03 19:27:07.949450679 +0000 UTC m=+37.003996515" Nov 03 19:27:08.924255 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.924219 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsx92" event={"ID":"abf3110c-9189-4738-ad6d-c70d1178ed91","Type":"ContainerStarted","Data":"b52bc0afe894fbd8a2163782e80bbfc6e407e7e30161fa4cc8971b37377907e7"} Nov 03 19:27:08.924650 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.924288 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsx92" event={"ID":"abf3110c-9189-4738-ad6d-c70d1178ed91","Type":"ContainerStarted","Data":"b4105a0824ab446282752da91ab1292aab4bbcf060a43315140128e258c859b2"} Nov 03 19:27:08.927654 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.927630 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzp5" event={"ID":"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea","Type":"ContainerStarted","Data":"8ee82f81c430fee8d93393ccb06746893ae23a7e2fb7a75b1b3cc3168abfd8d0"} Nov 03 19:27:08.927763 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.927693 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzp5" event={"ID":"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea","Type":"ContainerStarted","Data":"4066655a357695f9217269d3db8577602511229e2688e68f6c8857ca6ed431ac"} Nov 03 19:27:08.933757 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.933736 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" event={"ID":"0133e827-7681-4c7a-a5a8-34b9c5da874c","Type":"ContainerStarted","Data":"cf1cf408b49cde44e091dcd3bd19d81ae278601b93a4167390751d13d60ded7a"} Nov 03 19:27:08.943101 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.942847 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jsx92" podStartSLOduration=35.938374908 podStartE2EDuration="37.942834899s" totalImagesPullingTime="2.004459991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:08.942095059 +0000 UTC m=+37.996640915" watchObservedRunningTime="2026-11-03 19:27:08.942834899 +0000 UTC m=+37.997380740" Nov 03 19:27:08.961294 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.961242 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" podStartSLOduration=2.385282671 podStartE2EDuration="4.961224751s" totalImagesPullingTime="2.57594208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:08.960339244 +0000 UTC m=+38.014885089" watchObservedRunningTime="2026-11-03 19:27:08.961224751 +0000 UTC m=+38.015770595" Nov 03 19:27:08.983208 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:08.983169 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gdzp5" podStartSLOduration=2.927933035 podStartE2EDuration="4.983157743s" totalImagesPullingTime="1.968718768s" totalInitContainerRuntime="86.50594ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:08.981899228 +0000 UTC m=+38.036445090" watchObservedRunningTime="2026-11-03 19:27:08.983157743 +0000 UTC m=+38.037703584" Nov 03 19:27:09.640799 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.640765 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qf99w"] Nov 03 19:27:09.665842 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.665811 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qf99w"] Nov 03 19:27:09.666043 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.665924 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.679210 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.678300 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vvgzq\"" Nov 03 19:27:09.680564 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.679429 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 19:27:09.763934 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.763891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.764113 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.763954 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6lh\" (UniqueName: \"kubernetes.io/projected/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-kube-api-access-gj6lh\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.865426 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.865394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6lh\" (UniqueName: \"kubernetes.io/projected/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-kube-api-access-gj6lh\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.865594 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.865493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.868495 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.868469 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.884537 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.884493 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6lh\" (UniqueName: \"kubernetes.io/projected/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-kube-api-access-gj6lh\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:09.941274 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.941196 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" event={"ID":"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5","Type":"ContainerStarted","Data":"a4a8afa23d93492ad9a75aab34d6ef21ee9ca138fd38edde810e985e33b362f2"} Nov 03 19:27:09.941274 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.941273 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" event={"ID":"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5","Type":"ContainerStarted","Data":"43a00be00b2f5a7e0596cd826308dc53dffbe84a66000a4fe3d2c00ae6d14801"} Nov 03 19:27:09.941764 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.941293 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" event={"ID":"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5","Type":"ContainerStarted","Data":"1f5dac2de073d1f02c560cd7eabd6e5a1a91ee7ffe5039e171fc732692eac73b"} Nov 03 19:27:09.964270 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.964213 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" podStartSLOduration=2.839936555 podStartE2EDuration="5.964194587s" totalImagesPullingTime="3.124258032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:09.964035795 +0000 UTC m=+39.018581641" watchObservedRunningTime="2026-11-03 19:27:09.964194587 +0000 UTC m=+39.018740431" Nov 03 19:27:09.998770 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:09.998740 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:10.297886 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:10.297852 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qf99w"] Nov 03 19:27:10.303488 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:10.303461 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea05d8b_7c65_43b2_b74e_71b25b7f9da2.slice/crio-1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65 WatchSource:0}: Error finding container 1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65: Status 404 returned error can't find the container with id 1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65 Nov 03 19:27:10.945801 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:10.945760 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" event={"ID":"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2","Type":"ContainerStarted","Data":"1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65"} Nov 03 19:27:11.276500 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.276474 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:11.297339 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.295327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.303390 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.303343 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:11.311900 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.311880 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:27:11.312315 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.312296 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:27:11.312688 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.312670 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:27:11.312874 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.312857 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:27:11.319220 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.319195 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:27:11.319358 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.319341 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:27:11.383079 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.382926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383079 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.382999 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383079 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383353 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383353 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383353 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383353 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383400 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383424 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383541 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.383987 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.383725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388024 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388080 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388129 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388175 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388196 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388215 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388248 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388400 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-deig0nan6kf8q\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388405 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388463 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k9qt5\"" Nov 03 19:27:11.388553 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.388530 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:27:11.487390 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487390 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487390 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487421 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487526 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487644 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.487668 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487658 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487711 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487780 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488165 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.487846 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488597 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.488391 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488597 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.488501 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.488910 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.488886 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.490752 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.490725 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.490929 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.490908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.491527 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.491287 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.491612 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.491567 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.491712 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.491692 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.493928 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.492120 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.493928 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.493904 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.494336 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.494288 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.495438 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.495413 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.495523 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.495453 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.501015 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.500966 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.501479 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.501455 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.501479 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.501470 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.501629 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.501473 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.502335 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.502316 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.503833 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.503812 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.710243 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.710207 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:11.953340 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.953303 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" event={"ID":"a92b0d62-15bf-4769-8a0c-fa53b1079ca2","Type":"ContainerStarted","Data":"e7e8eb2025e65a05883633a71201d376ce6500feaa97c46f691fad76fda6adba"} Nov 03 19:27:11.953847 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.953392 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" event={"ID":"a92b0d62-15bf-4769-8a0c-fa53b1079ca2","Type":"ContainerStarted","Data":"229a0cd25b0c921d9c7390d8ee2d28654229ec2015f48865486d830fc12ca46a"} Nov 03 19:27:11.953847 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.953434 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" event={"ID":"a92b0d62-15bf-4769-8a0c-fa53b1079ca2","Type":"ContainerStarted","Data":"fd7deba7319323d28ab2d961a0ca20a7296a37976a80a1db28e6e8778e44d27b"} Nov 03 19:27:11.956662 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.955897 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dsngn" event={"ID":"11922a7d-e991-4de5-b28e-8ce492742213","Type":"ContainerStarted","Data":"6bef4fe10f850a4f25590411e340241859486b1849b79f3900a9940cd1d07c4c"} Nov 03 19:27:11.956662 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.956603 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:11.972687 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:11.972634 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dsngn" podStartSLOduration=35.744280715 podStartE2EDuration="40.972619186s" totalImagesPullingTime="5.228338471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:11.972460907 +0000 UTC m=+41.027006752" watchObservedRunningTime="2026-11-03 19:27:11.972619186 +0000 UTC m=+41.027165029" Nov 03 19:27:12.045552 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:12.045473 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:12.051917 ip-10-0-128-162 kubenswrapper[2563]: W1103 19:27:12.051883 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836e8d12_a2cc_45e7_8c70_70b11faf4d43.slice/crio-2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624 WatchSource:0}: Error finding container 2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624: Status 404 returned error can't find the container with id 2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624 Nov 03 19:27:12.961013 ip-10-0-128-162 kubenswrapper[2563]: I1103 19:27:12.960950 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624"} Nov 03 19:27:13.574739 ip-10-0-128-162 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:27:13.593394 ip-10-0-128-162 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:27:13.593570 ip-10-0-128-162 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:27:13.593619 ip-10-0-128-162 systemd[1]: kubelet.service: Consumed 2.968s CPU time. Nov 03 19:27:13.601633 ip-10-0-128-162 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:27:13.696298 ip-10-0-128-162 kubenswrapper[5233]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:27:13.696298 ip-10-0-128-162 kubenswrapper[5233]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:27:13.696298 ip-10-0-128-162 kubenswrapper[5233]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:27:13.696298 ip-10-0-128-162 kubenswrapper[5233]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:27:13.699481 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699384 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.699481 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699480 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.699481 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699486 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699493 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699498 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699503 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699508 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699512 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699517 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699521 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699525 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699529 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699533 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699538 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699548 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699552 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699556 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699560 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699564 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699568 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699572 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.699667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699577 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699581 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699585 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699590 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699595 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699599 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699603 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699607 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699611 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699616 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699621 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699625 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699629 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699634 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699639 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699643 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699647 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699651 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699655 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699659 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.700336 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699664 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699669 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699673 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699678 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699683 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699688 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699693 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699697 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699701 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699706 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699710 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699714 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699719 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699723 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699727 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699731 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699735 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699740 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699744 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.700942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699757 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699764 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699769 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699774 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699778 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699783 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699787 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699791 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699795 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699799 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699803 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699807 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699811 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699815 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699819 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699825 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699829 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699833 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699837 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699842 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.701577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699846 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699850 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699854 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699858 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699862 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699867 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699872 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699876 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699880 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699884 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699888 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699893 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699897 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699901 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699905 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699909 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699913 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699917 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699921 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699926 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.702243 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699930 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699934 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699938 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699942 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699947 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.699955 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701161 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701172 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701176 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701181 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701186 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701190 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701195 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701199 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701203 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701208 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701212 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701216 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701224 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.702874 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701230 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701234 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701239 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701244 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701249 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701254 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701258 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701262 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701266 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701270 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701274 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701278 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701282 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701286 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701290 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701294 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701298 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701302 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701306 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701310 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.703506 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701315 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701319 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701323 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701327 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701334 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701339 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701343 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701348 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701352 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701356 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701360 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701364 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701369 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701373 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701377 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701382 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701386 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701390 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701394 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701398 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.704181 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701403 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701407 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701411 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701415 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701421 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701425 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701429 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701433 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701437 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701441 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701445 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701449 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701453 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701458 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701462 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701466 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701472 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701476 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701480 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.704810 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701488 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701493 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701498 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701503 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701509 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701513 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701518 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701522 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701526 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701530 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701534 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701537 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701541 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701546 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701550 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701554 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701558 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701562 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701566 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701570 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.705482 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701574 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701578 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701582 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701586 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701591 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701595 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701599 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701603 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701607 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701612 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701616 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701620 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701624 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.701629 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702438 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702449 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702455 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702460 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702464 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702469 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.706199 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702473 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702477 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702481 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702485 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702490 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702494 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702498 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702502 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702506 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702510 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702514 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702518 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702522 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702527 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702531 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702535 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702540 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702544 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702548 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.706844 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702552 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702556 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702561 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702565 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702569 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702573 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702578 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702582 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702586 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702590 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702595 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702600 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702604 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702608 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702612 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702616 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702620 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702624 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702628 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.707427 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702632 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702636 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702640 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702644 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702648 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702652 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702656 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702660 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702664 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702668 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702672 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702676 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702680 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702684 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702688 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702692 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702696 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702700 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702705 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.708062 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702709 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702715 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702719 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702723 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702728 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702732 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702736 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702739 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702746 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702750 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702754 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702758 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702762 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702766 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702771 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702775 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702779 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702784 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702789 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702793 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.708712 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702803 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702808 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702812 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702816 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702820 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702824 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702831 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702836 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702841 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702845 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702849 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702853 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702857 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702861 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702866 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702870 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702874 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702881 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702888 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.709411 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702892 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702900 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702905 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.702909 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703046 5233 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703056 5233 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703068 5233 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703075 5233 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703082 5233 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703086 5233 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703093 5233 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703099 5233 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703104 5233 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703109 5233 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703114 5233 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703120 5233 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703125 5233 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703129 5233 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703134 5233 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703138 5233 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703143 5233 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703147 5233 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703155 5233 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703159 5233 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:27:13.710067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703165 5233 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703170 5233 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703175 5233 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703181 5233 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703186 5233 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703191 5233 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703197 5233 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703201 5233 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703206 5233 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703212 5233 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703217 5233 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703222 5233 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703229 5233 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703233 5233 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703239 5233 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703244 5233 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703249 5233 flags.go:64] FLAG: --enable-server="true" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703253 5233 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703260 5233 flags.go:64] FLAG: --event-burst="100" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703265 5233 flags.go:64] FLAG: --event-qps="50" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703270 5233 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703274 5233 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703280 5233 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703286 5233 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:27:13.711048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703290 5233 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703296 5233 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703301 5233 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703306 5233 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703311 5233 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703315 5233 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703320 5233 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703325 5233 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703329 5233 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703334 5233 flags.go:64] FLAG: --feature-gates="" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703340 5233 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703345 5233 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703350 5233 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703355 5233 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703360 5233 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703365 5233 flags.go:64] FLAG: --help="false" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703369 5233 flags.go:64] FLAG: --hostname-override="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703376 5233 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703381 5233 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703385 5233 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703391 5233 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703396 5233 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703402 5233 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:27:13.711854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703407 5233 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703411 5233 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703416 5233 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703421 5233 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703426 5233 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703430 5233 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703435 5233 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703439 5233 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703445 5233 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703450 5233 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703455 5233 flags.go:64] FLAG: --lock-file="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703459 5233 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703464 5233 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703468 5233 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703477 5233 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703481 5233 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703486 5233 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703490 5233 flags.go:64] FLAG: --logging-format="text" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703495 5233 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703500 5233 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703505 5233 flags.go:64] FLAG: --manifest-url="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703509 5233 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703516 5233 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703521 5233 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703527 5233 flags.go:64] FLAG: --max-pods="110" Nov 03 19:27:13.712698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703532 5233 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703538 5233 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703543 5233 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703547 5233 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703552 5233 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703557 5233 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703563 5233 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703574 5233 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703579 5233 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703584 5233 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703589 5233 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703593 5233 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703598 5233 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703602 5233 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703607 5233 flags.go:64] FLAG: --port="10250" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703611 5233 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703616 5233 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00c47fbc4173ac84e" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703621 5233 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703626 5233 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703630 5233 flags.go:64] FLAG: --register-node="true" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703635 5233 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703641 5233 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703645 5233 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703650 5233 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703654 5233 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703660 5233 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:27:13.713685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703665 5233 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703670 5233 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703674 5233 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703679 5233 flags.go:64] FLAG: --runonce="false" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703683 5233 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703689 5233 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703694 5233 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703703 5233 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703708 5233 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703713 5233 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703718 5233 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703724 5233 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703730 5233 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703734 5233 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703739 5233 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703743 5233 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703748 5233 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703753 5233 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703757 5233 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703766 5233 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703771 5233 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703775 5233 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703781 5233 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703786 5233 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703790 5233 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:27:13.714715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703795 5233 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703800 5233 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703805 5233 flags.go:64] FLAG: --v="2" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703815 5233 flags.go:64] FLAG: --version="false" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703821 5233 flags.go:64] FLAG: --vmodule="" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703827 5233 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.703832 5233 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704049 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704056 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704060 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704064 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704068 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704073 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704077 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704083 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704090 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704096 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704102 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704107 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704112 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704116 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.715539 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704121 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704125 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704129 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704133 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704137 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704142 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704146 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704150 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704155 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704158 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704163 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704167 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704171 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704175 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704179 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704183 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704187 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704192 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704196 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704200 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.716161 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704204 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704208 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704213 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704217 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704221 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704227 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704231 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704236 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704240 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704245 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704250 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704254 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704258 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704263 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704267 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704271 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704275 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704280 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704284 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704288 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.716755 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704292 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704296 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704299 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704304 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704308 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704312 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704316 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704320 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704324 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704329 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704333 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704337 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704341 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704345 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704349 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704352 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704356 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704362 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704367 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704372 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.717352 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704376 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704380 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704386 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704390 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704394 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704398 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704403 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704407 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704411 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704415 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704419 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704423 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704427 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704432 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704436 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704440 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704447 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704454 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704460 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.717952 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704465 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704471 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704475 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704480 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704485 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704489 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704494 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704498 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704502 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704506 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704511 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704516 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704520 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.704558 5233 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704611 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704617 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.718553 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704623 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704627 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704632 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704636 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704640 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704644 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704648 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704652 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704656 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704661 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704665 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704669 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704673 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704677 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704681 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704685 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704689 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704694 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704698 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704702 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.718993 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704707 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704712 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704716 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704720 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704724 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704728 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704734 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704738 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704742 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704747 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704751 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704755 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704761 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704765 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704769 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704773 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704777 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704781 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704785 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704790 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.719498 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704793 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704798 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704802 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704806 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704810 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704814 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704818 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704823 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704827 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704831 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704835 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704839 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704844 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704848 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704852 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704856 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704860 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704866 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704875 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.719998 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704881 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704887 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704893 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704897 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704902 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704910 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704914 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704919 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704923 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704927 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704931 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704935 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704940 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704945 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704949 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704953 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704957 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704961 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704965 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.720466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704983 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704988 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704992 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.704998 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705005 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705009 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705014 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705018 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705024 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705028 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705033 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705037 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705041 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705046 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705050 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705055 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705059 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705064 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705070 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.720943 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705076 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705080 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705084 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705088 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705092 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705096 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.705100 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.709343 5233 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.709357 5233 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.709375 5233 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.709380 5233 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709438 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709445 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709450 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709455 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709459 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709463 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709466 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709469 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.721487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709472 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709475 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709477 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709480 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709482 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709485 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709488 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709490 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709493 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709495 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709498 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709501 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709504 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709507 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709509 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709513 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709517 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709520 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709522 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709526 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.721938 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709530 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709536 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709540 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709544 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709547 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709550 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709552 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709555 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709557 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709560 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709563 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709565 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709568 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709570 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709573 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709575 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709578 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709580 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709583 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.722434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709585 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709588 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709590 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709593 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709595 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709598 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709601 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709604 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709606 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709609 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709611 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709614 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709616 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709619 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709622 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709627 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709631 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709636 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709643 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.722941 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709649 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709653 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709658 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709663 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709666 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709669 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709672 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709674 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709677 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709680 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709682 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709685 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709687 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709690 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709694 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709699 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709703 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709706 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709708 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.723415 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709711 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709714 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709717 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709720 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709723 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709726 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709730 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709735 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709739 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709742 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709745 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709749 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709753 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709758 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709761 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709763 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709766 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709769 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709771 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709774 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.723867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.709776 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.709819 5233 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710407 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710423 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710428 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710432 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710437 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710446 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710451 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710455 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710460 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710464 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710468 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710472 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710476 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710481 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.724449 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710485 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710490 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710494 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710499 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710508 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710512 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710517 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710522 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710526 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710533 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710539 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710544 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710549 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710554 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710558 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710563 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710573 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710577 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.724831 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710582 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710586 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710590 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710594 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710599 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710605 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710611 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710616 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710620 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710625 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710629 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710638 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710643 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710648 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710653 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710657 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710661 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710665 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710670 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.725276 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710674 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710678 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710683 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710687 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710697 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710701 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710706 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710710 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710715 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710719 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710723 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710728 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710732 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710737 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710741 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710745 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710753 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710758 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710762 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710770 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.725718 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710774 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710779 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710784 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710789 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710793 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710806 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710810 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710815 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710820 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710829 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710833 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710837 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710842 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710846 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710850 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710855 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710860 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710864 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710868 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.726211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710872 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710877 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710881 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710890 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710894 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710899 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710903 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710907 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710911 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710916 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710920 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710924 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710930 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710935 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710939 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.710945 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.711029 5233 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:27:13.726667 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711342 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711352 5233 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711358 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711363 5233 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711367 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711372 5233 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711375 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711382 5233 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711390 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711394 5233 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711397 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711399 5233 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711402 5233 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711405 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711407 5233 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711411 5233 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711414 5233 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711418 5233 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711423 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:27:13.727101 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711427 5233 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711432 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711437 5233 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711443 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711447 5233 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711451 5233 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711455 5233 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711459 5233 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711463 5233 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711469 5233 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711476 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711481 5233 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711485 5233 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711495 5233 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711500 5233 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711503 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711505 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711508 5233 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711511 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:27:13.727641 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711514 5233 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711516 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711520 5233 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711524 5233 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711529 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711534 5233 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711539 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711544 5233 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711549 5233 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711553 5233 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711557 5233 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711560 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711563 5233 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711568 5233 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711572 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711576 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711580 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711582 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711585 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:27:13.728152 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711588 5233 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711590 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711593 5233 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711595 5233 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711598 5233 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711601 5233 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711603 5233 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711608 5233 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711611 5233 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711615 5233 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711619 5233 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711624 5233 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711629 5233 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711634 5233 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711639 5233 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711643 5233 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711647 5233 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711652 5233 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711656 5233 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711660 5233 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711662 5233 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:27:13.728587 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711665 5233 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711667 5233 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711671 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711673 5233 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711676 5233 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711679 5233 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711681 5233 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711684 5233 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711688 5233 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711692 5233 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711696 5233 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711700 5233 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711703 5233 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711706 5233 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711709 5233 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711711 5233 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711714 5233 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711717 5233 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711723 5233 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711728 5233 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:27:13.729120 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711732 5233 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711741 5233 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711750 5233 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711755 5233 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711759 5233 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711762 5233 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711767 5233 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:13.711772 5233 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.711847 5233 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713150 5233 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713220 5233 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713408 5233 server.go:1014] "Starting client certificate rotation" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713541 5233 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713588 5233 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:21:31 +0000 UTC" deadline="2028-05-04 19:06:13.072955329 +0000 UTC" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713618 5233 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13151h38m59.359339641s" Nov 03 19:27:13.729908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.713844 5233 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:27:13.730455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.714882 5233 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:27:13.730455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.716145 5233 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:27:13.730455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.717956 5233 log.go:25] "Validated CRI v1 image API" Nov 03 19:27:13.730455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.718701 5233 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:27:13.730455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.720661 5233 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ba1ef60-4249-49c5-a883-35beef775cbe:/dev/nvme0n1p4 dc234ffa-a6e1-468f-8a94-7ea8cd497c24:/dev/nvme0n1p3] Nov 03 19:27:13.730654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.720679 5233 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624/userdata/shm major:0 minor:390 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b635637989e310c529c823dc2ca0fc8cbbb61d9dee2ba23e2e22204149dbea0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b635637989e310c529c823dc2ca0fc8cbbb61d9dee2ba23e2e22204149dbea0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f2438bb68eb6abf317a63fd27bcb696ecf96690f1e0f2898760efadd7c5c636/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f2438bb68eb6abf317a63fd27bcb696ecf96690f1e0f2898760efadd7c5c636/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c3dfdc1a4cd15b8c29e0a48ec030a7dbda7c98c9f01927dc508e37c13f5cd6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c3dfdc1a4cd15b8c29e0a48ec030a7dbda7c98c9f01927dc508e37c13f5cd6d/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1a394006db51797e72fde9e3c74f679ef276add1d254062abec54e5646d4bd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1a394006db51797e72fde9e3c74f679ef276add1d254062abec54e5646d4bd9/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~projected/kube-api-access-dsfch:{mountpoint:/var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~projected/kube-api-access-dsfch major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11922a7d-e991-4de5-b28e-8ce492742213/volumes/kubernetes.io~projected/kube-api-access-tgfkb:{mountpoint:/var/lib/kubelet/pods/11922a7d-e991-4de5-b28e-8ce492742213/volumes/kubernetes.io~projected/kube-api-access-tgfkb major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e9c8905-cc87-498c-8a3a-ec294b27fb9d/volumes/kubernetes.io~projected/kube-api-access-85lx9:{mountpoint:/var/lib/kubelet/pods/1e9c8905-cc87-498c-8a3a-ec294b27fb9d/volumes/kubernetes.io~projected/kube-api-access-85lx9 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e9c8905-cc87-498c-8a3a-ec294b27fb9d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/1e9c8905-cc87-498c-8a3a-ec294b27fb9d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21d46003-1ed7-40dd-b4b8-1e3a05448eee/volumes/kubernetes.io~projected/kube-api-access-m65xr:{mountpoint:/var/lib/kubelet/pods/21d46003-1ed7-40dd-b4b8-1e3a05448eee/volumes/kubernetes.io~projected/kube-api-access-m65xr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volumes/kubernetes.io~projected/kube-api-access-gl24g:{mountpoint:/var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volumes/kubernetes.io~projected/kube-api-access-gl24g major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4936d277-165f-4cf6-a9cb-6930c76e875a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4936d277-165f-4cf6-a9cb-6930c76e875a/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/volumes/kubernetes.io~projected/kube-api-access-gj6lh:{mountpoint:/var/lib/kubelet/pods/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/volumes/kubernetes.io~projected/kube-api-access-gj6lh major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5832527a-10bb-4d88-b8e8-c473df01d219/volumes/kubernetes.io~projected/kube-api-access-7l7m8:{mountpoint:/var/lib/kubelet/pods/5832527a-10bb-4d88-b8e8-c473df01d219/volumes/kubernetes.io~projected/kube-api-access-7l7m8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/652e8df9-bec9-4bdc-9535-4f2ea47f45d6/volumes/kubernetes.io~projected/kube-api-access-wqf89:{mountpoint:/var/lib/kubelet/pods/652e8df9-bec9-4bdc-9535-4f2ea47f45d6/volumes/kubernetes.io~projected/kube-api-access-wqf89 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/652e8df9-bec9-4bdc-9535-4f2ea47f45d6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/652e8df9-bec9-4bdc-9535-4f2ea47f45d6/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/704cd8b4-7baa-493a-9813-650106a42d13/volumes/kubernetes.io~projected/kube-api-access-8xrwr:{mountpoint:/var/lib/kubelet/pods/704cd8b4-7baa-493a-9813-650106a42d13/volumes/kubernetes.io~projected/kube-api-access-8xrwr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~empty-dir/config-out major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~projected/kube-api-access-6kdtz:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~projected/kube-api-access-6kdtz major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~projected/tls-assets major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/config major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/web-config major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89460d5b-9bf2-4029-9d64-1e21733554d0/volumes/kubernetes.io~projected/kube-api-access-nmmrn:{mountpoint:/var/lib/kubelet/pods/89460d5b-9bf2-4029-9d64-1e21733554d0/volumes/kubernetes.io~projected/kube-api-access-nmmrn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b2568c8-4309-407c-a381-83a1c8712e7d/volumes/kubernetes.io~projected/kube-api-access-vcklg:{mountpoint:/var/lib/kubelet/pods/9b2568c8-4309-407c-a381-83a1c8712e7d/volumes/kubernetes.io~projected/kube-api-access-vcklg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c908f51-a28e-46fb-af36-ac4cb9163671/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9c908f51-a28e-46fb-af36-ac4cb9163671/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~projected/kube-api-access-ssmhz:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~projected/kube-api-access-ssmhz major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abf3110c-9189-4738-ad6d-c70d1178ed91/volumes/kubernetes.io~projected/kube-api-access-f6862:{mountpoint:/var/lib/kubelet/pods/abf3110c-9189-4738-ad6d-c70d1178ed91/volumes/kubernetes.io~projected/kube-api-access-f6862 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abf3110c-9189-4738-ad6d-c70d1178ed91/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/abf3110c-9189-4738-ad6d-c70d1178ed91/volumes/kubernetes.io~secret/metrics-certs major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~projected/kube-api-access-xfkkf:{mountpoint:/var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~projected/kube-api-access-xfkkf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3132370-8820-4b1e-a9b2-adcd6fe1b958/volumes/kubernetes.io~projected/kube-api-access-zbnx8:{mountpoint:/var/lib/kubelet/pods/c3132370-8820-4b1e-a9b2-adcd6fe1b958/volumes/kubernetes.io~projected/kube-api-access-zbnx8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~projected/kube-api-access-d5r5f:{mountpoint:/var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~projected/kube-api-access-d5r5f major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1f99516-4d86-4bdd-8df3-39c1ec90f28a/volumes/kubernetes.io~projected/kube-api-access-nt4bb:{mountpoint:/var/lib/kubelet/pods/d1f99516-4d86-4bdd-8df3-39c1ec90f28a/volumes/kubernetes.io~projected/kube-api-access-nt4bb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1f99516-4d86-4bdd-8df3-39c1ec90f28a/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d1f99516-4d86-4bdd-8df3-39c1ec90f28a/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~projected/kube-api-access-5mfcg:{mountpoint:/var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~projected/kube-api-access-5mfcg major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ee27320a2039f61e4efc27523a702f40a80bbaff4cf4a267bc9a6d3746b335c9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/51b796519c2720722b46176533e60a3422ffef208bd835a266c18c1df857fd1d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/492d484df5231784bde77665ff99d7fc4750dda07ef12d8d0c15cf74432fa242/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/dd788f203e76f7283bb599cb539c8f6cf3187df63273a3c5411b455e1e6f5682/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1c116e7466f18951d348e21a1a93b7da322b6e2f39f700e25812aca05a98f156/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cf922a70f1ca3bb2e05132cdf63c25a8115b6b13686b18313f8f6a908f6c039c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e187fe40f903df3fdcf22f0f5541cf08a1f10a013114b0d60fcb1b61f8ca1b75/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b3cafa62a8466381e728213a9c40b4875cdb5e238b629cb39559235ab7bdde7c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d2d37929a31f435e860005c6978ab50142addea8243c176d043527024444f4ac/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/0cf17f9586d9ff5b05fd6e1dd6ccef046e4e3a3b540b16c1c2e4b0c26ceac985/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/5d37cb3b6f5b8e5e0cc30ea2be2878305b9c69c4ec5118e6980dcf068f63bbbe/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/9936b5759661bea132918a9b99d2625bcdef6afc557562a3c28344f32c3c7f96/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/dc373b4b5cb4052e16ddc68bf4388d14b94eb6bd5d20cc76ae0aa363e1f3fa6b/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f21d6c6354bdb74966e25df3c84fa92aec5b5f45576996ab97894b4d496a8484/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/547fc6c549e1c59d46b9dd50388a3f81130e0b731af608da3daa130320c8d775/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5bbb4b62eff7c5c754b51266956f045f80800d587f4e6adfac972331e5010dba/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b70faece8e91678507f75071a5e3897ddada887454eb29465081c59e09bb8d0d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4fdad5ee5d2cc4c066b62776d24e90dd27b1c07761d6bc3da1eba8f1bab2f475/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2390e8a327e061833bffa8e2bf019f3f404f6cb094edcbc0e6f2dac526c89160/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/23784439dcfa9f762737bd7474d4467f1f30b577c52eb08db6d7f7395f5ab854/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/02b170947b37edcb994470f0c1dbfb7c48f35d8a48401a1013dcd8aa824e594a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ed4fd9c3e75e81df83a045a89567b6da4ac7e90bd33579ac4ec2f96f12c78826/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2ed0edb844ad96df18a5863fec075f1aa5b9fe912d4eb0c8f92cba1dfd8b8543/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/a6d06a5aece4da151e531a6b5855f252dfbe2d89a13242509368389574cbf56f/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1aa57d65f87bfc7f1e96039e8452942951e63c7c77fcacaa92af857b61254d7f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/748884f2b5c665878175571f5fa49b6954bf483f4dc6d92df116a955cc6ec11f/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/1327288748c7dadb0542daef11f258ff26b1072b2efaf76bd88e17fb44171b76/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/14792fadfc38a77dc2296800fb1d5669030740d2b34a2dbe02a4a4b5c0dc20e6/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/cf0900adc91635c9f2238452fdf18b32bb6295c75b47ae80bdb02fbb8d682ed5/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/ddcb6dc24c7cfa628b7a65a7669d6e94ad5c7479bc9f5e12b7b2476877d22547/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/153c80c0a2a01b09fe7b87ee732ce3b66dd092528a82c33d2b61437aec28377d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2a74ff53942382465404ce9065348751c7cbb21013940a5a8ba115b5ca907b83/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/f4ee7ebf728aa4d3aa29e2c2c81f7241d0d2a79aedd36104b51d6d9803ba66bd/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c829884876c379abc286ad059392e98aafdb0273e15b197e572f28b7b47c2233/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b7a426a11e5a1d725bfb6841c1be4346851b370a60762e39cfa9533fac3c78f3/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/cb35ede2620ec279de647b8793c479d3f6e172ea96fd8437e8537de036418d22/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/066cb335164b1cf1170d55b594a9cd1909124a39643437e41a093267986c59cd/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/b1b500011854bd9eaf4867a20576bb98ff5c850f06e98614a2bb776c82e75efd/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/f8440fa57fa9699a9c333f45ce58406dc151affa2b33d5c86328b100fc428100/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/8e463b1ba0528234ac1ec55a7b19968a168639317dae1bdbcbb352c2269110f7/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c2069f0380922158dbd7a6590904322f02cf57a0b270b7802d86cbbfd6cbbf9d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/e951bf110ac23ead5a5f73ea8bae4b4059538868bfff40d5e9883b453dfa4616/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/9d6d4aba238da9cd21784af3793f4f140ccf163339563b3b2d237bb85941ddd5/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/69f95759883ed8d3eb6df6911d909dfacf5e859a0ee0d69b90165bbbafa4df0e/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/f42610bb06f46b1b3e8f1c7a6c46b37131e979337ac781a4ffb01ba00a00a840/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/c60176cb1f3e225d1dbcd0484d4b26b6cd602cafbc4d841a6e22bc095a4577a8/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/72153a029112445f3e20f05267561018abb8d0ff428611719f70c61a81c816ba/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/1233f60429a6634eee2e00abe0b4e6e2df8eda16582de42c4d718d0733cdc7fd/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/4904bd0de06851c5c55d79f653c57f5a49b3f0e11ad9c9ada2fe0f8634faf2bd/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/aff94ba80c3fd932fecb2771bd379cf0740d8f1e64f9f5bd3a527a6f1db92020/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/6511e1ec7c47d65566350edd0cc72bc9465b71c9d52c3793d9946cdb774b5dc0/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/f74453a4eb5d0e17849584c6a953589cd27e7e00a9df1af48db7877dd636d384/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/62b1aea5378950e09a23b915c7746480b247d9bdde705de226b8a39c52add239/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/7e6f5d9275f569aa9713a668b4e67f9c074265a0e70b845e1eed078a4f7667a2/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/f8d111d82e68f9d6b26626d140357436b5706894603d49cc2cabc7ddba131426/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/23a1e854b3d8eddff6b188280f711690b435503124b73046dc0016f367706770/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/af5e9ce111771f70af3e3f04f8102a16b5fd41c7d1b9b03fd7ec42debeade14b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/f2bfc111c9d00416c9aab1a301ed4f342e98fa2ba6c1789a472021384dbaae86/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/279987a53934d8f8a08cffc4f639d66047c636d3cbb1cc6637896e93e9817d1e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6d62fc8c34a45bb97e9dfadac7163879fb1119cf41952608c779051d7a53385f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/72ac9ca4790900f11ae709967a91cbe06f04624cdede913eba4b6374f4347180/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0f1028f3f62662cf17e05c8f54f0c204ece5ba4950e3ca5ebe476316e31f0d0b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0200c9298ce13dc421a077e7378fd6c68c6e9ad66763503f43f82640b17a2531/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/506e33cb7e76746c270c73cca3eb211f1f9c2340128545efa62d17aedd977b57/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0f323dd122d83263a623528414972cc2d2536a3371c9aae72ae9bdf6542443b3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/83a742947736ea4b409f941044a5da0ac9e8036db0f676705fb78905ce0ada6c/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6d526e864ca8cf3713bd986af30abe7a37b8185b7f2a5b93c79c9b66fed2a704/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/90c82dfa29df1b624a94eefa8a24a21e991edeb1ba8e64cddc187468b1c7e36f/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.728375 5233 manager.go:217] Machine: {Timestamp:2026-11-03 19:27:13.72769091 +0000 UTC m=+0.068804393 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101808 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274b1a6c8cb83012c96d50e7b5de4f SystemUUID:ec274b1a-6c8c-b830-12c9-6d50e7b5de4f BootID:44629097-52f3-4bb1-af5d-2697e1c56e91 Filesystems:[{Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f2bd992147ac176439026046fc563a6bf82f6311636cb5852d74a315c101179/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c34a1d4f0d12a05c975916b260d79b9487949a24ce1a0fddce3b3ae1b15fcc6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89460d5b-9bf2-4029-9d64-1e21733554d0/volumes/kubernetes.io~projected/kube-api-access-nmmrn DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e9c8905-cc87-498c-8a3a-ec294b27fb9d/volumes/kubernetes.io~projected/kube-api-access-85lx9 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volumes/kubernetes.io~projected/kube-api-access-gl24g DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e9c8905-cc87-498c-8a3a-ec294b27fb9d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:381 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1cd53d82eec43cc44a954d1dae5e9dec4d969013b049923688b6b53e4416bf4/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c3132370-8820-4b1e-a9b2-adcd6fe1b958/volumes/kubernetes.io~projected/kube-api-access-zbnx8 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21d46003-1ed7-40dd-b4b8-1e3a05448eee/volumes/kubernetes.io~projected/kube-api-access-m65xr DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/652e8df9-bec9-4bdc-9535-4f2ea47f45d6/volumes/kubernetes.io~projected/kube-api-access-wqf89 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~projected/kube-api-access-d5r5f DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c384144f7c7b802e6e32fcdd0b8774ef9e7a3979f73b3341d0528121387a722/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:380 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:383 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b635637989e310c529c823dc2ca0fc8cbbb61d9dee2ba23e2e22204149dbea0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76a17a51c69da3b9477a85b37a81aa85e3eb592f68881fca8562fc7abc8eb2e8/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc0f515fa114df57e6a2a8b54af8f5c872c170b1e27fa6f8d8d62f88c9c81902/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/974ae7f5da01c5cfd1ceed78172426426e0d178e2470a40adb4afbb0373bfed5/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/652e8df9-bec9-4bdc-9535-4f2ea47f45d6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4936d277-165f-4cf6-a9cb-6930c76e875a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624/userdata/shm DeviceMajor:0 DeviceMinor:390 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/volumes/kubernetes.io~projected/kube-api-access-5mfcg DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1f99516-4d86-4bdd-8df3-39c1ec90f28a/volumes/kubernetes.io~projected/kube-api-access-nt4bb DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11922a7d-e991-4de5-b28e-8ce492742213/volumes/kubernetes.io~projected/kube-api-access-tgfkb DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~projected/kube-api-access-6kdtz DeviceMajor:0 DeviceMinor:389 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2fc2ecf3608e70c8323b88befcd14e946947dc6ff3043239a3af31a9eec3c90/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f2438bb68eb6abf317a63fd27bcb696ecf96690f1e0f2898760efadd7c5c636/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a07282866d94c74b38a2f188dc08847597387afe3f6a0c97bb9510f60bcfe65/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c908f51-a28e-46fb-af36-ac4cb9163671/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2796abbc93a6b6533b8b6937e69ffdb3e16603c370e9db651a010fbedcc7b491/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5832527a-10bb-4d88-b8e8-c473df01d219/volumes/kubernetes.io~projected/kube-api-access-7l7m8 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4d35a83792b9bceaaa87c8fd06e586f6dc01eb3a60b3124c32532a9ac0576d5/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05493f7d6ab23953e8fe2dbed59de547eb6968a8e7c54325f2a21c0fa5e4bf4e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1a394006db51797e72fde9e3c74f679ef276add1d254062abec54e5646d4bd9/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:379 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b2568c8-4309-407c-a381-83a1c8712e7d/volumes/kubernetes.io~projected/kube-api-access-vcklg DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b69aca2ac988bd99ac9ce2d4ece17be1148482ea5f4279e44cc3c5cf560a9867/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0bc14f6d0616af1fabae701a835e2f9c3957ec5d1f01bb49568665e2270f986/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0133e827-7681-4c7a-a5a8-34b9c5da874c/volumes/kubernetes.io~projected/kube-api-access-dsfch DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1f99516-4d86-4bdd-8df3-39c1ec90f28a/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/volumes/kubernetes.io~projected/kube-api-access-gj6lh DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abf3110c-9189-4738-ad6d-c70d1178ed91/volumes/kubernetes.io~projected/kube-api-access-f6862 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b22777af-ae5b-4635-a4ff-140ae4be22e9/volumes/kubernetes.io~projected/kube-api-access-xfkkf DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c3dfdc1a4cd15b8c29e0a48ec030a7dbda7c98c9f01927dc508e37c13f5cd6d/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a92b0d62-15bf-4769-8a0c-fa53b1079ca2/volumes/kubernetes.io~projected/kube-api-access-ssmhz DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abf3110c-9189-4738-ad6d-c70d1178ed91/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:384 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/704cd8b4-7baa-493a-9813-650106a42d13/volumes/kubernetes.io~projected/kube-api-access-8xrwr DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fe747e4-481a-461c-9b8a-a85e9f4957d5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edc0d313fce6153f81a2c613cc0c055518a8c171b53fcfaf49354a884c5a2c3f/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05493f7d6ab2395 MacAddress:ba:1b:85:de:19:82 Speed:10000 Mtu:8901} {Name:1a07282866d94c7 MacAddress:3a:2a:90:d1:fe:f0 Speed:10000 Mtu:8901} {Name:2c01436f6340623 MacAddress:3e:b9:9e:ff:16:18 Speed:10000 Mtu:8901} {Name:2f2bd992147ac17 MacAddress:ee:64:0c:50:c8:ef Speed:10000 Mtu:8901} {Name:5c384144f7c7b80 MacAddress:5e:df:eb:93:d8:fb Speed:10000 Mtu:8901} {Name:9c3dfdc1a4cd15b MacAddress:9e:f1:3a:78:7b:10 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f0:ed:cf:03 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:c4:7b:a8:98:5f Speed:0 Mtu:8901} {Name:c1cd53d82eec43c MacAddress:7a:ae:f6:54:46:37 Speed:10000 Mtu:8901} {Name:d1a394006db5179 MacAddress:26:10:13:7a:c2:fa Speed:10000 Mtu:8901} {Name:edc0d313fce6153 MacAddress:82:de:13:cd:85:4e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f0:ed:cf:03 Speed:0 Mtu:9001} {Name:f4d35a83792b9bc MacAddress:ee:2b:b6:f0:47:e1 Speed:10000 Mtu:8901} {Name:fc0f515fa114df5 MacAddress:fe:f4:5f:a5:3b:69 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:92:41:d4:ab:af:f5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:2f:d4:5f:10:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.728838 5233 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.728883 5233 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729450 5233 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729480 5233 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-162.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729798 5233 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729811 5233 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729825 5233 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729852 5233 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.729994 5233 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730547 5233 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730657 5233 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730690 5233 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730700 5233 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730709 5233 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730717 5233 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:27:13.730966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.730724 5233 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:27:13.731678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.731380 5233 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:27:13.731678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.731390 5233 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:27:13.731945 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.731927 5233 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:27:13.750209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.750154 5233 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:27:13.754025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.754011 5233 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:27:13.785213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785189 5233 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:27:13.785213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785205 5233 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:27:13.785367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785332 5233 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:27:13.785567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785551 5233 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:27:13.785628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785592 5233 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:27:13.785749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785734 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:27:13.785803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785759 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:27:13.785803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785770 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:27:13.785803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785780 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:27:13.785803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785788 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:27:13.785803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785797 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785806 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785815 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785827 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785836 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785849 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785865 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785903 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:27:13.786073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.785913 5233 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:27:13.789425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.789408 5233 server.go:1263] "Started kubelet" Nov 03 19:27:13.789517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.789471 5233 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:27:13.789727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.789572 5233 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:27:13.789866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.789851 5233 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:27:13.790235 ip-10-0-128-162 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:27:13.790333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.790239 5233 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:27:13.791313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.791296 5233 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794414 5233 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794544 5233 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794602 5233 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:21:33 +0000 UTC" deadline="2028-07-16 09:23:51.027023868 +0000 UTC" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794628 5233 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14893h56m37.232398177s" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794828 5233 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794837 5233 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:27:13.795243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.794930 5233 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:27:13.795728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.795405 5233 factory.go:55] Registering systemd factory Nov 03 19:27:13.795728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.795423 5233 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797138 5233 factory.go:153] Registering CRI-O factory Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797157 5233 factory.go:223] Registration of the crio container factory successfully Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797521 5233 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797555 5233 factory.go:103] Registering Raw factory Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797571 5233 manager.go:1199] Started watching for new ooms in manager Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797816 5233 manager.go:319] Starting recovery of all containers Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797895 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9c8905-cc87-498c-8a3a-ec294b27fb9d" volumeName="kubernetes.io/empty-dir/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-data-volume" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797934 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797950 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.797963 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf3110c-9189-4738-ad6d-c70d1178ed91" volumeName="kubernetes.io/projected/abf3110c-9189-4738-ad6d-c70d1178ed91-kube-api-access-f6862" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798003 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5" volumeName="kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798018 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798031 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf3110c-9189-4738-ad6d-c70d1178ed91" volumeName="kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798044 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/empty-dir/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-tmp" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798059 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" volumeName="kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798071 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5" volumeName="kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798085 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d46003-1ed7-40dd-b4b8-1e3a05448eee" volumeName="kubernetes.io/configmap/21d46003-1ed7-40dd-b4b8-1e3a05448eee-serviceca" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798098 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe747e4-481a-461c-9b8a-a85e9f4957d5" volumeName="kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798109 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe747e4-481a-461c-9b8a-a85e9f4957d5" volumeName="kubernetes.io/secret/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798125 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798139 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798150 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3132370-8820-4b1e-a9b2-adcd6fe1b958" volumeName="kubernetes.io/projected/c3132370-8820-4b1e-a9b2-adcd6fe1b958-kube-api-access-zbnx8" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798164 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" volumeName="kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798176 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89460d5b-9bf2-4029-9d64-1e21733554d0" volumeName="kubernetes.io/projected/89460d5b-9bf2-4029-9d64-1e21733554d0-kube-api-access-nmmrn" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798188 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0133e827-7681-4c7a-a5a8-34b9c5da874c" volumeName="kubernetes.io/configmap/0133e827-7681-4c7a-a5a8-34b9c5da874c-metrics-client-ca" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798199 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0133e827-7681-4c7a-a5a8-34b9c5da874c" volumeName="kubernetes.io/projected/0133e827-7681-4c7a-a5a8-34b9c5da874c-kube-api-access-dsfch" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798211 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0133e827-7681-4c7a-a5a8-34b9c5da874c" volumeName="kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798222 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9c8905-cc87-498c-8a3a-ec294b27fb9d" volumeName="kubernetes.io/configmap/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798233 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652e8df9-bec9-4bdc-9535-4f2ea47f45d6" volumeName="kubernetes.io/projected/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-kube-api-access-wqf89" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798261 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798272 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798284 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798295 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798307 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798552 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22777af-ae5b-4635-a4ff-140ae4be22e9" volumeName="kubernetes.io/projected/b22777af-ae5b-4635-a4ff-140ae4be22e9-kube-api-access-xfkkf" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798577 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3132370-8820-4b1e-a9b2-adcd6fe1b958" volumeName="kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798595 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" volumeName="kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-metrics-client-ca" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798604 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798612 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798620 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-grpc-tls" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798628 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798638 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe747e4-481a-461c-9b8a-a85e9f4957d5" volumeName="kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798646 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11922a7d-e991-4de5-b28e-8ce492742213" volumeName="kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798654 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea05d8b-7c65-43b2-b74e-71b25b7f9da2" volumeName="kubernetes.io/secret/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798663 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652e8df9-bec9-4bdc-9535-4f2ea47f45d6" volumeName="kubernetes.io/secret/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-cert" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798674 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe747e4-481a-461c-9b8a-a85e9f4957d5" volumeName="kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-env-overrides" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798699 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="704cd8b4-7baa-493a-9813-650106a42d13" volumeName="kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-cni-binary-copy" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798708 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798716 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798728 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798738 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798746 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3132370-8820-4b1e-a9b2-adcd6fe1b958" volumeName="kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798755 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3132370-8820-4b1e-a9b2-adcd6fe1b958" volumeName="kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-binary-copy" seLinuxMountContext="" Nov 03 19:27:13.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798774 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe747e4-481a-461c-9b8a-a85e9f4957d5" volumeName="kubernetes.io/projected/3fe747e4-481a-461c-9b8a-a85e9f4957d5-kube-api-access-gl24g" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798782 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5832527a-10bb-4d88-b8e8-c473df01d219" volumeName="kubernetes.io/projected/5832527a-10bb-4d88-b8e8-c473df01d219-kube-api-access-7l7m8" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798793 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/configmap/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-metrics-client-ca" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798810 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1f99516-4d86-4bdd-8df3-39c1ec90f28a" volumeName="kubernetes.io/projected/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-kube-api-access-nt4bb" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798823 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5" volumeName="kubernetes.io/projected/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-api-access-5mfcg" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798832 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798844 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9c8905-cc87-498c-8a3a-ec294b27fb9d" volumeName="kubernetes.io/projected/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-api-access-85lx9" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798854 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798866 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89460d5b-9bf2-4029-9d64-1e21733554d0" volumeName="kubernetes.io/empty-dir/89460d5b-9bf2-4029-9d64-1e21733554d0-tmp-dir" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798890 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22777af-ae5b-4635-a4ff-140ae4be22e9" volumeName="kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-tuned" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798898 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5" volumeName="kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798905 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5" volumeName="kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-metrics-client-ca" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798915 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="704cd8b4-7baa-493a-9813-650106a42d13" volumeName="kubernetes.io/projected/704cd8b4-7baa-493a-9813-650106a42d13-kube-api-access-8xrwr" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798922 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798929 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b2568c8-4309-407c-a381-83a1c8712e7d" volumeName="kubernetes.io/projected/9b2568c8-4309-407c-a381-83a1c8712e7d-kube-api-access-vcklg" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798938 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1f99516-4d86-4bdd-8df3-39c1ec90f28a" volumeName="kubernetes.io/configmap/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-config-volume" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798945 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1f99516-4d86-4bdd-8df3-39c1ec90f28a" volumeName="kubernetes.io/empty-dir/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-tmp-dir" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798952 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1f99516-4d86-4bdd-8df3-39c1ec90f28a" volumeName="kubernetes.io/secret/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-metrics-tls" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798960 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798980 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea05d8b-7c65-43b2-b74e-71b25b7f9da2" volumeName="kubernetes.io/projected/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-kube-api-access-gj6lh" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.798993 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799004 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799017 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" volumeName="kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-tls" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799030 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a92b0d62-15bf-4769-8a0c-fa53b1079ca2" volumeName="kubernetes.io/projected/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-kube-api-access-ssmhz" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799039 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9c8905-cc87-498c-8a3a-ec294b27fb9d" volumeName="kubernetes.io/secret/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799050 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="704cd8b4-7baa-493a-9813-650106a42d13" volumeName="kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-multus-daemon-config" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799061 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799076 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5" volumeName="kubernetes.io/empty-dir/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-volume-directive-shadow" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799087 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836e8d12-a2cc-45e7-8c70-70b11faf4d43" volumeName="kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799095 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c908f51-a28e-46fb-af36-ac4cb9163671" volumeName="kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799105 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d46003-1ed7-40dd-b4b8-1e3a05448eee" volumeName="kubernetes.io/projected/21d46003-1ed7-40dd-b4b8-1e3a05448eee-kube-api-access-m65xr" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799115 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4936d277-165f-4cf6-a9cb-6930c76e875a" volumeName="kubernetes.io/configmap/4936d277-165f-4cf6-a9cb-6930c76e875a-konnectivity-ca" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799124 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22777af-ae5b-4635-a4ff-140ae4be22e9" volumeName="kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-tmp" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799136 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" volumeName="kubernetes.io/projected/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-kube-api-access-d5r5f" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799147 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0133e827-7681-4c7a-a5a8-34b9c5da874c" volumeName="kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799155 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4936d277-165f-4cf6-a9cb-6930c76e875a" volumeName="kubernetes.io/secret/4936d277-165f-4cf6-a9cb-6930c76e875a-agent-certs" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799163 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5832527a-10bb-4d88-b8e8-c473df01d219" volumeName="kubernetes.io/configmap/5832527a-10bb-4d88-b8e8-c473df01d219-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799171 5233 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" volumeName="kubernetes.io/empty-dir/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-textfile" seLinuxMountContext="" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799179 5233 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.799184 5233 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:27:13.801230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.801135 5233 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:27:13.803898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.803881 5233 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:27:13.811603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.811576 5233 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:27:13.828492 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.828466 5233 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:27:13.828492 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.828489 5233 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:27:13.828642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.828507 5233 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:27:13.828642 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:13.828549 5233 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:27:13.832583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.832555 5233 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:27:13.838344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.838302 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="f59aa3333c649833f3c6b6e9999ecfcc37fac80ce04d795558b41cc59d88a209" exitCode=0 Nov 03 19:27:13.838344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.838337 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="2862140de656b528eef3f1e4b55765f9e54853425991825c943d7915c9b84d15" exitCode=0 Nov 03 19:27:13.838555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.838351 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="e2b02e35f1fe1c1bd7d047f3e89a5a40da5ee13778bc1d524108b680d97747c1" exitCode=0 Nov 03 19:27:13.838555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.838362 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="459284b853a35aca369122c330b45d8288d8be3b6e5c8dc0a2bb835742b31003" exitCode=0 Nov 03 19:27:13.838555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.838375 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="63d0796e4b04aef9e9f53c84cfe4ff687bcde3b4392b59993677216edc8c258e" exitCode=0 Nov 03 19:27:13.838555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.838387 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3132370-8820-4b1e-a9b2-adcd6fe1b958" containerID="94fdfef6fce65119bc341c4c0d878d286de85164328dbbc86888b4f28d4ebfb8" exitCode=0 Nov 03 19:27:13.853696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.853673 5233 generic.go:356] "Generic (PLEG): container finished" podID="c87c6d25-883d-4f0d-b4d3-8c77abbb41ea" containerID="3feabb6ded2c6f35b21ca700d3b89f23bd925303c16e7836a2fad7c80d907558" exitCode=0 Nov 03 19:27:13.875921 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.875890 5233 generic.go:356] "Generic (PLEG): container finished" podID="852f7206562983f9f71af157dcc57010" containerID="d42a014f266ee361e442a8c7fb0b3a97b0f166d34e9860edaf1194a7cddec1df" exitCode=0 Nov 03 19:27:13.904753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.904734 5233 manager.go:324] Recovery completed Nov 03 19:27:13.927280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927259 5233 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:27:13.927280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927275 5233 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927293 5233 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927407 5233 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927418 5233 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927433 5233 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927439 5233 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927444 5233 policy_none.go:50] "Start" Nov 03 19:27:13.927462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927463 5233 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:27:13.928315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927472 5233 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:27:13.928315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927594 5233 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:27:13.928315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927601 5233 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:27:13.928315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.927606 5233 policy_none.go:44] "Start" Nov 03 19:27:13.928809 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:13.928790 5233 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 19:27:13.932724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.932142 5233 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:27:13.938003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.937987 5233 manager.go:341] "Starting Device Plugin manager" Nov 03 19:27:13.938085 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:13.938019 5233 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:27:13.938085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938031 5233 server.go:87] "Starting device plugin registration server" Nov 03 19:27:13.938365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938353 5233 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:27:13.938395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938367 5233 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:27:13.938461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938444 5233 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:27:13.938581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938545 5233 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:27:13.938686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938590 5233 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:27:13.938686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938597 5233 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:27:13.938791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.938658 5233 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:27:13.938575736Z","UUID":"d6138194-7616-4e43-a3bf-2aaafa032fb1","Handler":null,"Name":"","Endpoint":""} Nov 03 19:27:13.939947 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.939929 5233 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:27:13.940044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:13.939959 5233 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:27:13.940816 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:13.940795 5233 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:27:14.039258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.039194 5233 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:27:14.040504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.040485 5233 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:27:14.040612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.040514 5233 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:27:14.040612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.040524 5233 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-162.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:27:14.040612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.040563 5233 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.053740 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.053720 5233 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.053803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.053773 5233 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.129040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.128997 5233 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal"] Nov 03 19:27:14.129638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129622 5233 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.129745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129632 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"9c8881dae007849f880e1281962f0bcb","Type":"ContainerStarted","Data":"6c09dcb24bcad9a60a4d97609cd7f7b2e283f805d065b6a0b5a0d2ecba39351c"} Nov 03 19:27:14.129745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129690 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"9c8881dae007849f880e1281962f0bcb","Type":"ContainerStarted","Data":"0d899a1176810ad38e4ca30ad500af1f005e064c8212851834f323b8fdc0219c"} Nov 03 19:27:14.129745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129715 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"852f7206562983f9f71af157dcc57010","Type":"ContainerStarted","Data":"d76d4097a855ffb08087e9b2efcb71f1828f3479585d85e9870e6295e718d181"} Nov 03 19:27:14.129745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129729 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"852f7206562983f9f71af157dcc57010","Type":"ContainerDied","Data":"d42a014f266ee361e442a8c7fb0b3a97b0f166d34e9860edaf1194a7cddec1df"} Nov 03 19:27:14.129745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129745 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" event={"ID":"852f7206562983f9f71af157dcc57010","Type":"ContainerStarted","Data":"677535b51f2af8d57b944a4883a3dee5a757cd777db67006ee33f114ceda4eab"} Nov 03 19:27:14.129959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.129868 5233 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.148639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.148620 5233 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:27:14.148707 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:14.148660 5233 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.151793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.151779 5233 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:27:14.151830 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:14.151813 5233 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.206127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.206105 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.206238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.206152 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.206238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.206180 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c8881dae007849f880e1281962f0bcb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"9c8881dae007849f880e1281962f0bcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.306845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.306781 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.306845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.306819 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.306845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.306839 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c8881dae007849f880e1281962f0bcb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"9c8881dae007849f880e1281962f0bcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.307037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.306884 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.307037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.306892 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/852f7206562983f9f71af157dcc57010-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"852f7206562983f9f71af157dcc57010\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.307037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.306928 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c8881dae007849f880e1281962f0bcb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" (UID: \"9c8881dae007849f880e1281962f0bcb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.731221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.731192 5233 apiserver.go:52] "Watching apiserver" Nov 03 19:27:14.748802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.748777 5233 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:27:14.749383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.749361 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-npmtd","openshift-dns/dns-default-7fzhd","openshift-multus/multus-additional-cni-plugins-m8xhh","openshift-multus/network-metrics-daemon-jsx92","openshift-network-diagnostics/network-check-target-dsngn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk","kube-system/global-pull-secret-syncer-s9hlp","openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm","openshift-multus/multus-vf8zn","openshift-cluster-node-tuning-operator/tuned-8x5vv","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/monitoring-plugin-77f4f76594-qf99w","openshift-monitoring/node-exporter-gdzp5","kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal","kube-system/konnectivity-agent-f5dt8","openshift-monitoring/thanos-querier-7f68bfd554-gj6f6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-6k5vq","openshift-dns/node-resolver-dmtq4","openshift-image-registry/node-ca-22xws","openshift-ingress-canary/ingress-canary-xtdsf","openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5","openshift-network-operator/iptables-alerter-x2xzc"] Nov 03 19:27:14.756429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.756399 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.756539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.756459 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.756737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.756720 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.756904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.756842 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.756904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.756873 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.756904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.756893 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.759341 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.759317 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:27:14.759341 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.759332 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.760190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.760176 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:27:14.762378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.762360 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.764833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.764818 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:27:14.765779 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.765764 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:27:14.768524 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.768511 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.770304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770283 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770303 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770384 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770394 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770389 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770411 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770413 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.770437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770433 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:27:14.770700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770573 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:27:14.770824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770803 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:27:14.770824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.770813 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:27:14.772337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.772323 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.772896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.772884 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:27:14.775176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.775160 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.775883 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.775866 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.776004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.775987 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.776084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776017 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.776084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776018 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:27:14.776084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776052 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:27:14.776235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776127 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:27:14.776750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776734 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 19:27:14.776750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776750 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:27:14.776864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.776821 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:27:14.779598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.779584 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:27:14.779958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.779926 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:27:14.780019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.780005 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:27:14.785645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.785626 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:27:14.786379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.786355 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:27:14.787367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.787342 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:27:14.787786 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.787767 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 19:27:14.788025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.788008 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:27:14.788122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.788055 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:27:14.788198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.788177 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:27:14.788298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.788279 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 19:27:14.788497 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.788479 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:27:14.788566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.788550 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:27:14.789116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.789098 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 19:27:14.789331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.789317 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xv5d7\"" Nov 03 19:27:14.789396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.789317 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:27:14.789448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.789435 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:27:14.790767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.790747 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:27:14.790887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.790874 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 19:27:14.790942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.790878 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 19:27:14.791019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.791003 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:27:14.791073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.791006 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:27:14.791073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.791045 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:27:14.791151 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.791077 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vvgzq\"" Nov 03 19:27:14.792411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792399 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lz2t\"" Nov 03 19:27:14.792449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792404 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8cx9\"" Nov 03 19:27:14.792490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792478 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:27:14.792522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792498 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lsdlf\"" Nov 03 19:27:14.792635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792620 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:27:14.792635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792633 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xp5ps\"" Nov 03 19:27:14.792771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792644 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5vlcb\"" Nov 03 19:27:14.792771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792663 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hwgf7\"" Nov 03 19:27:14.792771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792686 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:27:14.792771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792696 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6kk8\"" Nov 03 19:27:14.792771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792766 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pv4cw\"" Nov 03 19:27:14.792965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792800 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 19:27:14.792965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792895 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-deig0nan6kf8q\"" Nov 03 19:27:14.792965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792904 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xlp4b\"" Nov 03 19:27:14.792965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792913 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6b5k\"" Nov 03 19:27:14.792965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792938 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:27:14.792965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792961 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5m7m493jtqc6c\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792961 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6b2s9\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792991 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.792994 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793010 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793024 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5zzs7\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793031 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gtgvx\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793050 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5w6kx\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793055 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793066 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmsjd\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793068 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qhgjf\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793083 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-st58b\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793087 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s8pjd\"" Nov 03 19:27:14.793195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.793119 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 19:27:14.794160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.794148 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:27:14.794680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.794668 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:27:14.794792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.794780 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:27:14.794888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.794876 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k9qt5\"" Nov 03 19:27:14.796647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.796632 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:27:14.797826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.797809 5233 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:27:14.809507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809489 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-socket-dir-parent\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.809600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809512 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-netns\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.809600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809537 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-tls\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.809600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809554 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4936d277-165f-4cf6-a9cb-6930c76e875a-konnectivity-ca\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:27:14.809600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809571 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-var-lib-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.809765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809625 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysconfig\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.809765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809701 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.809885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809865 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrwr\" (UniqueName: \"kubernetes.io/projected/704cd8b4-7baa-493a-9813-650106a42d13-kube-api-access-8xrwr\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.809941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809886 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4936d277-165f-4cf6-a9cb-6930c76e875a-konnectivity-ca\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:27:14.809941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809898 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85lx9\" (UniqueName: \"kubernetes.io/projected/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-api-access-85lx9\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.809941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809916 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:14.809941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809865 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-tls\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.809941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809936 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-slash\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.810323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.809965 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.810323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810040 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.810323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810095 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.810323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810173 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.810323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810212 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf3110c-9189-4738-ad6d-c70d1178ed91-metrics-certs\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:14.810323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810277 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.810491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810343 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.810491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810371 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-modprobe-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.810491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810395 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.810491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810463 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-var-lib-kubelet\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.810600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810497 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-metrics-tls\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.810600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810525 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.810600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810550 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.810719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810619 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.810719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810647 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-metrics-tls\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.810719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810668 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.810807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810745 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.810807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810750 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-sys-fs\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.810885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810867 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4bb\" (UniqueName: \"kubernetes.io/projected/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-kube-api-access-nt4bb\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.810922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810875 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.810922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810906 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-host\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.810922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810917 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810927 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-kubelet\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.811035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.810962 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.811035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811022 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.811119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811097 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811125 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811152 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-kubernetes\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.811212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811180 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.811212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811176 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkkf\" (UniqueName: \"kubernetes.io/projected/b22777af-ae5b-4635-a4ff-140ae4be22e9-kube-api-access-xfkkf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.811275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811220 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-log-socket\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.811275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811250 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811274 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-crio-socket\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.811342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811275 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811312 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21d46003-1ed7-40dd-b4b8-1e3a05448eee-serviceca\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.811342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811333 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811407 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-lib-modules\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.811461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811451 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811506 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.811572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811535 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.811632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811615 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89460d5b-9bf2-4029-9d64-1e21733554d0-hosts-file\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.811688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811646 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89460d5b-9bf2-4029-9d64-1e21733554d0-tmp-dir\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.811741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811669 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-ovn\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.811741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811712 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-grpc-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.811825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811743 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-cni-binary-copy\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.811825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811755 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.811825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811761 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89460d5b-9bf2-4029-9d64-1e21733554d0-tmp-dir\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.811825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811778 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811850 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811895 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-config\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811937 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-cni-binary-copy\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811944 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811979 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-grpc-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.811996 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0133e827-7681-4c7a-a5a8-34b9c5da874c-kube-api-access-dsfch\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812015 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-metrics-client-ca\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.812026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812030 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-multus-daemon-config\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812047 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812048 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21d46003-1ed7-40dd-b4b8-1e3a05448eee-serviceca\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812093 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-etc-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812121 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-node-log\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812130 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812166 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfcg\" (UniqueName: \"kubernetes.io/projected/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-api-access-5mfcg\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812184 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-config\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812185 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812200 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-hostroot\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812226 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/704cd8b4-7baa-493a-9813-650106a42d13-multus-daemon-config\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812227 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-conf-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812237 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-metrics-client-ca\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812280 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-tuned\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812334 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-tmp-dir\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.812398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812352 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812415 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-tmp-dir\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812404 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812459 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-sys\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812473 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-tuned\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812475 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812499 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-systemd\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812517 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-kubelet\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812533 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcklg\" (UniqueName: \"kubernetes.io/projected/9b2568c8-4309-407c-a381-83a1c8712e7d-kube-api-access-vcklg\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812547 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-data-volume\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812560 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21d46003-1ed7-40dd-b4b8-1e3a05448eee-host\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812573 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-tmp\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812612 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmrn\" (UniqueName: \"kubernetes.io/projected/89460d5b-9bf2-4029-9d64-1e21733554d0-kube-api-access-nmmrn\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812656 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812672 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-data-volume\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812663 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812694 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812704 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b22777af-ae5b-4635-a4ff-140ae4be22e9-tmp\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812748 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812782 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812802 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-bin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812834 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812850 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7m8\" (UniqueName: \"kubernetes.io/projected/5832527a-10bb-4d88-b8e8-c473df01d219-kube-api-access-7l7m8\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812865 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812907 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3132370-8820-4b1e-a9b2-adcd6fe1b958-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812909 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-tls\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812962 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-env-overrides\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.812997 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.813030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813015 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-dbus\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813055 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813118 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813156 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-wtmp\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813191 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813168 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813235 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6862\" (UniqueName: \"kubernetes.io/projected/abf3110c-9189-4738-ad6d-c70d1178ed91-kube-api-access-f6862\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813253 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-env-overrides\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813265 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovn-node-metrics-cert\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813290 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-config-volume\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813315 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnx8\" (UniqueName: \"kubernetes.io/projected/c3132370-8820-4b1e-a9b2-adcd6fe1b958-kube-api-access-zbnx8\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813343 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-device-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813371 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813478 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-textfile\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813510 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-netns\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813538 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-netd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813541 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovn-node-metrics-cert\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813567 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-textfile\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813586 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813611 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-run\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813628 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-systemd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813647 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-config-volume\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813675 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-system-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813693 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-cnibin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813707 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-socket-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813725 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813741 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813743 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl24g\" (UniqueName: \"kubernetes.io/projected/3fe747e4-481a-461c-9b8a-a85e9f4957d5-kube-api-access-gl24g\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813768 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-kubelet-config\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813837 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813883 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-etc-kubernetes\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813928 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.813936 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814010 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814027 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814040 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-conf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.814103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814066 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-tmp\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814192 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814227 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814253 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-tmp\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814257 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814328 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0133e827-7681-4c7a-a5a8-34b9c5da874c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814372 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6lh\" (UniqueName: \"kubernetes.io/projected/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-kube-api-access-gj6lh\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814488 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4936d277-165f-4cf6-a9cb-6930c76e875a-agent-certs\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814543 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814622 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-system-cni-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814649 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-multus\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814670 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-multus-certs\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814670 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4936d277-165f-4cf6-a9cb-6930c76e875a-agent-certs\") pod \"konnectivity-agent-f5dt8\" (UID: \"4936d277-165f-4cf6-a9cb-6930c76e875a\") " pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814708 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5832527a-10bb-4d88-b8e8-c473df01d219-host-slash\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814745 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-script-lib\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814764 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814792 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqf89\" (UniqueName: \"kubernetes.io/projected/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-kube-api-access-wqf89\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.814910 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmhz\" (UniqueName: \"kubernetes.io/projected/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-kube-api-access-ssmhz\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815009 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5832527a-10bb-4d88-b8e8-c473df01d219-iptables-alerter-script\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815008 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815092 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815107 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fe747e4-481a-461c-9b8a-a85e9f4957d5-ovnkube-script-lib\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815126 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815152 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815176 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-cert\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815204 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0133e827-7681-4c7a-a5a8-34b9c5da874c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815243 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5832527a-10bb-4d88-b8e8-c473df01d219-iptables-alerter-script\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815251 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-root\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815309 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815329 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815358 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65xr\" (UniqueName: \"kubernetes.io/projected/21d46003-1ed7-40dd-b4b8-1e3a05448eee-kube-api-access-m65xr\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815381 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-sys\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815400 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-bin\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815418 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815423 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-k8s-cni-cncf-io\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815485 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5r5f\" (UniqueName: \"kubernetes.io/projected/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-kube-api-access-d5r5f\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815508 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815519 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815540 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-cert\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:14.815550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815559 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815613 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0133e827-7681-4c7a-a5a8-34b9c5da874c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815681 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815703 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-metrics-client-ca\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815720 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815734 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-systemd-units\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815748 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cnibin\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815763 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815766 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815799 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815817 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-registration-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815858 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c908f51-a28e-46fb-af36-ac4cb9163671-original-pull-secret\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815870 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815899 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815913 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-metrics-client-ca\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815924 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815954 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815998 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.815962 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.816092 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.816163 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.816894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.816182 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.821664 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.821642 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4bb\" (UniqueName: \"kubernetes.io/projected/d1f99516-4d86-4bdd-8df3-39c1ec90f28a-kube-api-access-nt4bb\") pod \"dns-default-7fzhd\" (UID: \"d1f99516-4d86-4bdd-8df3-39c1ec90f28a\") " pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:14.821724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.821670 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85lx9\" (UniqueName: \"kubernetes.io/projected/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-kube-api-access-85lx9\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.824421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.824405 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkkf\" (UniqueName: \"kubernetes.io/projected/b22777af-ae5b-4635-a4ff-140ae4be22e9-kube-api-access-xfkkf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.825198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.825179 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfcg\" (UniqueName: \"kubernetes.io/projected/ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5-kube-api-access-5mfcg\") pod \"kube-state-metrics-5dc8d7d89b-z69vm\" (UID: \"ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z69vm" Nov 03 19:27:14.826773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.826751 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") pod \"prometheus-k8s-0\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:14.829866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.829844 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfkb\" (UniqueName: \"kubernetes.io/projected/11922a7d-e991-4de5-b28e-8ce492742213-kube-api-access-tgfkb\") pod \"network-check-target-dsngn\" (UID: \"11922a7d-e991-4de5-b28e-8ce492742213\") " pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:14.830424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.830398 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfch\" (UniqueName: \"kubernetes.io/projected/0133e827-7681-4c7a-a5a8-34b9c5da874c-kube-api-access-dsfch\") pod \"openshift-state-metrics-7bd7489957-zc8b5\" (UID: \"0133e827-7681-4c7a-a5a8-34b9c5da874c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zc8b5" Nov 03 19:27:14.830692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.830677 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmrn\" (UniqueName: \"kubernetes.io/projected/89460d5b-9bf2-4029-9d64-1e21733554d0-kube-api-access-nmmrn\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.830735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.830700 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnx8\" (UniqueName: \"kubernetes.io/projected/c3132370-8820-4b1e-a9b2-adcd6fe1b958-kube-api-access-zbnx8\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.830991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.830964 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65xr\" (UniqueName: \"kubernetes.io/projected/21d46003-1ed7-40dd-b4b8-1e3a05448eee-kube-api-access-m65xr\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.833403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.833384 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrwr\" (UniqueName: \"kubernetes.io/projected/704cd8b4-7baa-493a-9813-650106a42d13-kube-api-access-8xrwr\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.835406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.835389 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7m8\" (UniqueName: \"kubernetes.io/projected/5832527a-10bb-4d88-b8e8-c473df01d219-kube-api-access-7l7m8\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.842655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.842167 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcklg\" (UniqueName: \"kubernetes.io/projected/9b2568c8-4309-407c-a381-83a1c8712e7d-kube-api-access-vcklg\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.842655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.842302 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6lh\" (UniqueName: \"kubernetes.io/projected/4ea05d8b-7c65-43b2-b74e-71b25b7f9da2-kube-api-access-gj6lh\") pod \"monitoring-plugin-77f4f76594-qf99w\" (UID: \"4ea05d8b-7c65-43b2-b74e-71b25b7f9da2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:14.842655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.842521 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl24g\" (UniqueName: \"kubernetes.io/projected/3fe747e4-481a-461c-9b8a-a85e9f4957d5-kube-api-access-gl24g\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.842655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.842616 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6862\" (UniqueName: \"kubernetes.io/projected/abf3110c-9189-4738-ad6d-c70d1178ed91-kube-api-access-f6862\") pod \"network-metrics-daemon-jsx92\" (UID: \"abf3110c-9189-4738-ad6d-c70d1178ed91\") " pod="openshift-multus/network-metrics-daemon-jsx92" Nov 03 19:27:14.842901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.842736 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmhz\" (UniqueName: \"kubernetes.io/projected/a92b0d62-15bf-4769-8a0c-fa53b1079ca2-kube-api-access-ssmhz\") pod \"thanos-querier-7f68bfd554-gj6f6\" (UID: \"a92b0d62-15bf-4769-8a0c-fa53b1079ca2\") " pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:14.842901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.842886 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5r5f\" (UniqueName: \"kubernetes.io/projected/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-kube-api-access-d5r5f\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.843394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.843373 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqf89\" (UniqueName: \"kubernetes.io/projected/652e8df9-bec9-4bdc-9535-4f2ea47f45d6-kube-api-access-wqf89\") pod \"ingress-canary-xtdsf\" (UID: \"652e8df9-bec9-4bdc-9535-4f2ea47f45d6\") " pod="openshift-ingress-canary/ingress-canary-xtdsf" Nov 03 19:27:14.895470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.895443 5233 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.895611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.895578 5233 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.908212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.908193 5233 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:27:14.908310 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:14.908242 5233 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.909345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.909331 5233 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:27:14.909410 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:14.909365 5233 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-162.us-east-2.compute.internal" Nov 03 19:27:14.916627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916607 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-kubernetes\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.916711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916633 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-log-socket\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916649 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-crio-socket\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.916711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916665 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-lib-modules\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.916711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916679 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89460d5b-9bf2-4029-9d64-1e21733554d0-hosts-file\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.916711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916694 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-ovn\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916716 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916734 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-etc-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916762 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-kubernetes\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916784 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-node-log\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916754 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-ovn\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916790 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e9c8905-cc87-498c-8a3a-ec294b27fb9d-crio-socket\") pod \"insights-runtime-extractor-npmtd\" (UID: \"1e9c8905-cc87-498c-8a3a-ec294b27fb9d\") " pod="openshift-insights/insights-runtime-extractor-npmtd" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916809 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916825 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-node-log\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916828 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89460d5b-9bf2-4029-9d64-1e21733554d0-hosts-file\") pod \"node-resolver-dmtq4\" (UID: \"89460d5b-9bf2-4029-9d64-1e21733554d0\") " pod="openshift-dns/node-resolver-dmtq4" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916836 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-etc-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916745 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-log-socket\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916857 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-hostroot\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916887 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-conf-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916916 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-sys\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916928 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-hostroot\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916887 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-lib-modules\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.916950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916940 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-systemd\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916965 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-sys\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916995 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-systemd\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.916941 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-conf-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917005 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-kubelet\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917041 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21d46003-1ed7-40dd-b4b8-1e3a05448eee-host\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917063 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-kubelet\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917072 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-bin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917091 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21d46003-1ed7-40dd-b4b8-1e3a05448eee-host\") pod \"node-ca-22xws\" (UID: \"21d46003-1ed7-40dd-b4b8-1e3a05448eee\") " pod="openshift-image-registry/node-ca-22xws" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917097 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917093 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-bin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917124 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917152 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-dbus\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917169 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917183 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-wtmp\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917211 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-device-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917219 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-dbus\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917229 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-netns\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917252 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-netns\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917256 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-netd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917290 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-run\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917293 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-netd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917300 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-node-exporter-wtmp\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917294 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-device-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917315 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-systemd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917331 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-system-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917340 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-run\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917348 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-cnibin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917351 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-systemd\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917372 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-socket-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917327 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917388 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-cnibin\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917394 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-system-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917401 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-kubelet-config\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917429 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-etc-kubernetes\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917449 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c908f51-a28e-46fb-af36-ac4cb9163671-kubelet-config\") pod \"global-pull-secret-syncer-s9hlp\" (UID: \"9c908f51-a28e-46fb-af36-ac4cb9163671\") " pod="kube-system/global-pull-secret-syncer-s9hlp" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917454 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-conf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917515 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-system-cni-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917539 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-conf\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917542 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-multus\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917459 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-etc-kubernetes\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917430 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-socket-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917563 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-system-cni-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917577 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-multus-certs\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917582 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-var-lib-cni-multus\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917598 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-multus-certs\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917605 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5832527a-10bb-4d88-b8e8-c473df01d219-host-slash\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917638 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-root\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917663 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-sys\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.917642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917663 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5832527a-10bb-4d88-b8e8-c473df01d219-host-slash\") pod \"iptables-alerter-x2xzc\" (UID: \"5832527a-10bb-4d88-b8e8-c473df01d219\") " pod="openshift-network-operator/iptables-alerter-x2xzc" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917691 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c87c6d25-883d-4f0d-b4d3-8c77abbb41ea-root\") pod \"node-exporter-gdzp5\" (UID: \"c87c6d25-883d-4f0d-b4d3-8c77abbb41ea\") " pod="openshift-monitoring/node-exporter-gdzp5" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917685 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-bin\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917714 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-sys\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917730 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-cni-bin\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917741 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-k8s-cni-cncf-io\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917769 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917795 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917804 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-k8s-cni-cncf-io\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917818 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-systemd-units\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917841 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917845 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-systemd-units\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917847 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cnibin\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917888 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-registration-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917890 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3132370-8820-4b1e-a9b2-adcd6fe1b958-cnibin\") pod \"multus-additional-cni-plugins-m8xhh\" (UID: \"c3132370-8820-4b1e-a9b2-adcd6fe1b958\") " pod="openshift-multus/multus-additional-cni-plugins-m8xhh" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917914 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917924 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-registration-dir\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917935 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysctl-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917949 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-socket-dir-parent\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917961 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-cni-dir\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.917985 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-netns\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918010 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-var-lib-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918031 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysconfig\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918031 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-host-run-netns\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918031 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/704cd8b4-7baa-493a-9813-650106a42d13-multus-socket-dir-parent\") pod \"multus-vf8zn\" (UID: \"704cd8b4-7baa-493a-9813-650106a42d13\") " pod="openshift-multus/multus-vf8zn" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918061 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-slash\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918085 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918097 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-slash\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918080 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-sysconfig\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918060 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-var-lib-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918110 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-modprobe-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918135 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918159 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-var-lib-kubelet\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918183 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-sys-fs\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918201 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-etc-modprobe-d\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918206 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-host\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918227 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-kubelet\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918241 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918252 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-var-lib-kubelet\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918260 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b2568c8-4309-407c-a381-83a1c8712e7d-sys-fs\") pod \"aws-ebs-csi-driver-node-vzstk\" (UID: \"9b2568c8-4309-407c-a381-83a1c8712e7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzstk" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918282 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-host-kubelet\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918288 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22777af-ae5b-4635-a4ff-140ae4be22e9-host\") pod \"tuned-8x5vv\" (UID: \"b22777af-ae5b-4635-a4ff-140ae4be22e9\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5vv" Nov 03 19:27:14.919273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:14.918299 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fe747e4-481a-461c-9b8a-a85e9f4957d5-run-openvswitch\") pod \"ovnkube-node-6k5vq\" (UID: \"3fe747e4-481a-461c-9b8a-a85e9f4957d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:15.063158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:15.063135 5233 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:27:15.251482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:15.251455 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:15.253090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:15.253068 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7fzhd" Nov 03 19:27:15.902926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:15.902838 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" exitCode=0 Nov 03 19:27:15.902926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:15.902909 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} Nov 03 19:27:16.071434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:16.071408 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:16.075744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:16.075723 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qf99w" Nov 03 19:27:17.418130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:17.418100 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:17.425168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:17.425139 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f68bfd554-gj6f6" Nov 03 19:27:18.919624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:18.919518 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} Nov 03 19:27:18.920091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:18.919633 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} Nov 03 19:27:18.920091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:18.919679 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} Nov 03 19:27:19.918721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:19.918694 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:19.926055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:19.925957 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} Nov 03 19:27:19.926055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:19.926022 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} Nov 03 19:27:19.926055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:19.926046 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerStarted","Data":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} Nov 03 19:27:20.544279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:20.544247 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:21.276079 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:21.276042 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:27:23.930382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:23.930350 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:24.045915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:24.045882 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:24.046182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:24.046165 5233 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:27:24.167097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:24.167069 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:24.185404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:24.185334 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:24.958379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:24.958353 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:38.119490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:38.119459 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f5dt8" Nov 03 19:27:43.932907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:43.932872 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dsngn" Nov 03 19:27:44.299186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:44.299154 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6k5vq" Nov 03 19:27:51.276059 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:51.276026 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:27:54.008811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.008686 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:54.009502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.009452 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="prometheus" containerID="cri-o://57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" gracePeriod=600 Nov 03 19:27:54.010105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.009808 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" gracePeriod=600 Nov 03 19:27:54.010105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.009829 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy" containerID="cri-o://468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" gracePeriod=600 Nov 03 19:27:54.010105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.009829 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="thanos-sidecar" containerID="cri-o://836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" gracePeriod=600 Nov 03 19:27:54.010105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.009894 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-web" containerID="cri-o://75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" gracePeriod=600 Nov 03 19:27:54.011162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.011108 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="config-reloader" containerID="cri-o://b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" gracePeriod=600 Nov 03 19:27:54.266306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.266255 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:54.425450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425347 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425498 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425532 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425552 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425576 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425596 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425618 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425653 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425686 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425722 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425748 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425777 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425808 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425836 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425880 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.425936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425911 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.425960 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426015 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426042 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426074 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426106 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") pod \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\" (UID: \"836e8d12-a2cc-45e7-8c70-70b11faf4d43\") " Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426334 5233 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.426425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426348 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.426786 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.426444 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.427829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.427313 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:54.428157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.428126 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.428379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.428354 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.428468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.428407 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.428533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.428512 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:54.428940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.428872 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.429153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.429018 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.429153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.429037 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:54.429290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.429171 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.429290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.429236 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.429409 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.429333 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "kube-api-access-6kdtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:54.430361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.430340 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:54.430676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.430656 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.431160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.431127 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.431248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.431181 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.431300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.431246 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.440159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.440138 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config" pod "836e8d12-a2cc-45e7-8c70-70b11faf4d43" (UID: "836e8d12-a2cc-45e7-8c70-70b11faf4d43"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:54.527371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527300 5233 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527336 5233 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-web-config\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527348 5233 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527358 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kdtz\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-kube-api-access-6kdtz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527367 5233 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527377 5233 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config-out\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527385 5233 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-kube-rbac-proxy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527394 5233 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527403 5233 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-db\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527412 5233 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527420 5233 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-config\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527429 5233 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527437 5233 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527445 5233 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-metrics-client-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527453 5233 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-metrics-client-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527462 5233 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/836e8d12-a2cc-45e7-8c70-70b11faf4d43-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527470 5233 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/836e8d12-a2cc-45e7-8c70-70b11faf4d43-tls-assets\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527478 5233 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:54.527643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:54.527488 5233 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/836e8d12-a2cc-45e7-8c70-70b11faf4d43-secret-grpc-tls\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:55.054089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054057 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" exitCode=0 Nov 03 19:27:55.054089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054079 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" exitCode=0 Nov 03 19:27:55.054089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054085 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" exitCode=0 Nov 03 19:27:55.054089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054091 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" exitCode=0 Nov 03 19:27:55.054089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054097 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" exitCode=0 Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054103 5233 generic.go:356] "Generic (PLEG): container finished" podID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" exitCode=0 Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054129 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054148 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054198 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054230 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054258 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054278 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054282 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054312 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} Nov 03 19:27:55.054766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.054340 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"836e8d12-a2cc-45e7-8c70-70b11faf4d43","Type":"ContainerDied","Data":"2c01436f63406233c16b6255d0147d348736f766a94ae02afb9e915fd4c6f624"} Nov 03 19:27:55.062685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.062664 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.070051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.070034 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.076332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.076317 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.080814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.080787 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:55.083075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.083060 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.086273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.086254 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:55.089787 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.089762 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.096403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.096389 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.102504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.102485 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.102782 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.102762 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.102842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.102791 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} err="failed to get container status \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" Nov 03 19:27:55.102842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.102820 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.103085 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.103071 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.103135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103089 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} err="failed to get container status \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" Nov 03 19:27:55.103135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103101 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.103351 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.103333 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.103426 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103358 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} err="failed to get container status \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" Nov 03 19:27:55.103426 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103372 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.103623 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.103609 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.103661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103625 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} err="failed to get container status \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" Nov 03 19:27:55.103661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103637 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.103850 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.103834 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.103887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103855 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} err="failed to get container status \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" Nov 03 19:27:55.103887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.103868 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.104354 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.104338 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.104411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.104358 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} err="failed to get container status \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" Nov 03 19:27:55.104411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.104373 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.104620 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:27:55.104605 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.104655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.104626 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} err="failed to get container status \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" Nov 03 19:27:55.104655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.104638 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.104866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.104849 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} err="failed to get container status \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" Nov 03 19:27:55.104908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.104870 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.105098 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105081 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} err="failed to get container status \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" Nov 03 19:27:55.105137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105098 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.105297 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105281 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} err="failed to get container status \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" Nov 03 19:27:55.105334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105297 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.105504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105489 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} err="failed to get container status \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" Nov 03 19:27:55.105546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105503 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.105704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105689 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} err="failed to get container status \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" Nov 03 19:27:55.105744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105704 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.105899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105883 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} err="failed to get container status \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" Nov 03 19:27:55.105942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.105899 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.106123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106108 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} err="failed to get container status \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" Nov 03 19:27:55.106176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106123 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.106330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106313 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} err="failed to get container status \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" Nov 03 19:27:55.106371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106331 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.106675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106654 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} err="failed to get container status \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" Nov 03 19:27:55.106675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106671 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.106881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106858 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} err="failed to get container status \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" Nov 03 19:27:55.106881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.106878 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.107155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107125 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} err="failed to get container status \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" Nov 03 19:27:55.107213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107155 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.107464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107445 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} err="failed to get container status \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" Nov 03 19:27:55.107464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107462 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.107690 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107674 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} err="failed to get container status \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" Nov 03 19:27:55.107746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107690 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.107926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107906 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} err="failed to get container status \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" Nov 03 19:27:55.107926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.107925 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.108144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108128 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} err="failed to get container status \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" Nov 03 19:27:55.108187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108144 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.108329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108314 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} err="failed to get container status \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" Nov 03 19:27:55.108369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108330 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.108523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108507 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} err="failed to get container status \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" Nov 03 19:27:55.108523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108521 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.108717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108700 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} err="failed to get container status \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" Nov 03 19:27:55.108758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108717 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.108898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108883 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} err="failed to get container status \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" Nov 03 19:27:55.108937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.108898 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.109153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109132 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} err="failed to get container status \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" Nov 03 19:27:55.109153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109153 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.109361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109345 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} err="failed to get container status \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" Nov 03 19:27:55.109412 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109361 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.109592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109577 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} err="failed to get container status \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" Nov 03 19:27:55.109633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109592 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.109796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109779 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} err="failed to get container status \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" Nov 03 19:27:55.109836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.109796 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.110017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110001 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} err="failed to get container status \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" Nov 03 19:27:55.110017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110017 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.110226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110211 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} err="failed to get container status \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" Nov 03 19:27:55.110271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110226 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.110410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110394 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} err="failed to get container status \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" Nov 03 19:27:55.110453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110409 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.110624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110608 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} err="failed to get container status \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" Nov 03 19:27:55.110624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110623 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.110806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110789 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} err="failed to get container status \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" Nov 03 19:27:55.110849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.110805 5233 scope.go:122] "RemoveContainer" containerID="a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c" Nov 03 19:27:55.111030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111012 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c"} err="failed to get container status \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": rpc error: code = NotFound desc = could not find container \"a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c\": container with ID starting with a8ce21b068768c4a8290c197f07804ecd7d57604543a64f4a2a5ea67d2bf7a7c not found: ID does not exist" Nov 03 19:27:55.111079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111032 5233 scope.go:122] "RemoveContainer" containerID="468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23" Nov 03 19:27:55.111270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111249 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23"} err="failed to get container status \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": rpc error: code = NotFound desc = could not find container \"468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23\": container with ID starting with 468e23636c381c4567b8352022b6ccd7b27429a7d7b1433582b4249596ec6f23 not found: ID does not exist" Nov 03 19:27:55.111313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111272 5233 scope.go:122] "RemoveContainer" containerID="75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f" Nov 03 19:27:55.111461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111445 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f"} err="failed to get container status \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": rpc error: code = NotFound desc = could not find container \"75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f\": container with ID starting with 75dcdaa143eef8554ae6345f68c54e40d706007bf6d7c2308deb67ab8961524f not found: ID does not exist" Nov 03 19:27:55.111503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111461 5233 scope.go:122] "RemoveContainer" containerID="836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab" Nov 03 19:27:55.111641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111627 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab"} err="failed to get container status \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": rpc error: code = NotFound desc = could not find container \"836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab\": container with ID starting with 836a3e8d4b55b22f954ac75b39f0a556b5f9aba3aaa2b79a8d72da6deee936ab not found: ID does not exist" Nov 03 19:27:55.111641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111640 5233 scope.go:122] "RemoveContainer" containerID="b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0" Nov 03 19:27:55.111847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111829 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0"} err="failed to get container status \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": rpc error: code = NotFound desc = could not find container \"b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0\": container with ID starting with b3656ec859ebd68117a9fa5a0df7eedee8a6c40836fe938a186327c05dc439d0 not found: ID does not exist" Nov 03 19:27:55.111885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.111847 5233 scope.go:122] "RemoveContainer" containerID="57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a" Nov 03 19:27:55.112097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.112078 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a"} err="failed to get container status \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": rpc error: code = NotFound desc = could not find container \"57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a\": container with ID starting with 57bf71b5950db611efd026ca7d707966ca9cdf11c5f27ab11de6490ae78ee33a not found: ID does not exist" Nov 03 19:27:55.112148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.112098 5233 scope.go:122] "RemoveContainer" containerID="61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557" Nov 03 19:27:55.112302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.112280 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557"} err="failed to get container status \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": rpc error: code = NotFound desc = could not find container \"61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557\": container with ID starting with 61db143eec96812c81d9a454e10765ca498513bd4416483fd9a1a50f59e71557 not found: ID does not exist" Nov 03 19:27:55.117033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117012 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:55.117294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117279 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="thanos-sidecar" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117297 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="thanos-sidecar" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117309 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117315 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117325 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-thanos" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117332 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-thanos" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117355 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="prometheus" Nov 03 19:27:55.117369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117362 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="prometheus" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117379 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="config-reloader" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117387 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="config-reloader" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117399 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="init-config-reloader" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117405 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="init-config-reloader" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117417 5233 cpu_manager.go:414] "containerMap: removing container" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-web" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117424 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-web" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117493 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="thanos-sidecar" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117504 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117512 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-thanos" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117524 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="prometheus" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117535 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="config-reloader" Nov 03 19:27:55.117706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.117545 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" containerName="kube-rbac-proxy-web" Nov 03 19:27:55.120987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.120957 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.126202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126178 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:27:55.126305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126289 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:27:55.126388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126304 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:27:55.126388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126316 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:27:55.126502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126394 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:27:55.126502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126414 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:27:55.126502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.126475 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:27:55.127570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.127553 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:27:55.127570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.127566 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:27:55.127682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.127568 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:27:55.128241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.128222 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:27:55.128318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.128228 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k9qt5\"" Nov 03 19:27:55.128318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.128265 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:27:55.128318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.128265 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-deig0nan6kf8q\"" Nov 03 19:27:55.129906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.129891 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:27:55.138001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.137983 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:27:55.138086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.137982 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:27:55.140482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.140462 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:55.231839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.231790 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.231839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.231843 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.231862 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.231922 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb066071-701f-4f31-84b0-f1e4bbb906eb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232022 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232054 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232078 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-config\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232154 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232180 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb066071-701f-4f31-84b0-f1e4bbb906eb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232251 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232272 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232326 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232345 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232365 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232380 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232394 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smj8j\" (UniqueName: \"kubernetes.io/projected/cb066071-701f-4f31-84b0-f1e4bbb906eb-kube-api-access-smj8j\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232493 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232521 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232536 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.232605 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.232560 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333204 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333237 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333255 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333373 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333416 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333443 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333468 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smj8j\" (UniqueName: \"kubernetes.io/projected/cb066071-701f-4f31-84b0-f1e4bbb906eb-kube-api-access-smj8j\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333629 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333644 5233 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:27:55.333760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333673 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333700 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333733 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333774 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333815 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333842 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333873 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb066071-701f-4f31-84b0-f1e4bbb906eb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333903 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333924 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.333958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.333948 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-config\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.334330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.334016 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.334330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.334028 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.334330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.334042 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb066071-701f-4f31-84b0-f1e4bbb906eb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.334330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.334179 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.335085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.335055 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.336796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.336466 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.336796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.336479 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb066071-701f-4f31-84b0-f1e4bbb906eb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.336796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.336554 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.336796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.336710 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.336796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.336779 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.337805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.337255 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.337805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.337263 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.337805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.337711 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.337805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.337724 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.337805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.337732 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.338119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.337934 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb066071-701f-4f31-84b0-f1e4bbb906eb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.338321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.338302 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb066071-701f-4f31-84b0-f1e4bbb906eb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.338929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.338908 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.339301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.339278 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.339603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.339585 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-config\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.339660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.339649 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb066071-701f-4f31-84b0-f1e4bbb906eb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.346361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.346344 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smj8j\" (UniqueName: \"kubernetes.io/projected/cb066071-701f-4f31-84b0-f1e4bbb906eb-kube-api-access-smj8j\") pod \"prometheus-k8s-0\" (UID: \"cb066071-701f-4f31-84b0-f1e4bbb906eb\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.436948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.436920 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:27:55.725458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.725428 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:27:55.728612 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:27:55.728583 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb066071_701f_4f31_84b0_f1e4bbb906eb.slice/crio-7866dd6bffa92e241ef6e1361412e96cc2c180cdc36bb37c2370450bdd15546d WatchSource:0}: Error finding container 7866dd6bffa92e241ef6e1361412e96cc2c180cdc36bb37c2370450bdd15546d: Status 404 returned error can't find the container with id 7866dd6bffa92e241ef6e1361412e96cc2c180cdc36bb37c2370450bdd15546d Nov 03 19:27:55.833461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:55.833433 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836e8d12-a2cc-45e7-8c70-70b11faf4d43" path="/var/lib/kubelet/pods/836e8d12-a2cc-45e7-8c70-70b11faf4d43/volumes" Nov 03 19:27:56.060332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:56.060256 5233 generic.go:356] "Generic (PLEG): container finished" podID="cb066071-701f-4f31-84b0-f1e4bbb906eb" containerID="205fc97a5167154a95154f92d02b356a355ab64ccdb189483e2d9344aeb89051" exitCode=0 Nov 03 19:27:56.060332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:56.060295 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerDied","Data":"205fc97a5167154a95154f92d02b356a355ab64ccdb189483e2d9344aeb89051"} Nov 03 19:27:56.060685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:56.060348 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"7866dd6bffa92e241ef6e1361412e96cc2c180cdc36bb37c2370450bdd15546d"} Nov 03 19:27:57.071528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.071492 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"e61258a7d48f4076b98f72ad6c37d86e7332b9cc674eaada61c14b0ac76668c6"} Nov 03 19:27:57.071922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.071580 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"a5876a03f56ddac8f712aa53c44e09fd7608c9acf8fb2ef14f53faa4fc6a3866"} Nov 03 19:27:57.071922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.071608 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"0447f7230128b6a94027b2ba95f409ab570866c497b6521ac8590748600fb893"} Nov 03 19:27:57.071922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.071632 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"dfe9e06415f2330cd3bfdc77afed7bb5c815fa8d41f953bfa71d871e84057df8"} Nov 03 19:27:57.071922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.071656 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"cbb8f6a984ec4b1e0418b423bd4fe5419a48d2bd5981fd81eecb2e77fc720897"} Nov 03 19:27:57.071922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.071679 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb066071-701f-4f31-84b0-f1e4bbb906eb","Type":"ContainerStarted","Data":"a22833a5150b90774021072eef3a8530fd6b3f19f643b28eaf1de4d5718e6ea9"} Nov 03 19:27:57.108989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:27:57.108921 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.093739423 podStartE2EDuration="2.10890983s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.170407ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:57.107326083 +0000 UTC m=+43.448439568" watchObservedRunningTime="2026-11-03 19:27:57.10890983 +0000 UTC m=+43.450023335" Nov 03 19:28:00.438016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:28:00.437950 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:28:21.274687 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:28:21.274655 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:28:44.233763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:28:44.233726 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:28:44.248598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:28:44.248574 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:28:45.252296 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:28:45.252270 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:28:51.276036 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:28:51.275997 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:29:21.275169 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:29:21.275129 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:29:51.275386 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:29:51.275309 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:30:21.275427 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:30:21.275389 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:30:51.275155 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:30:51.275113 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:31:21.275202 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:31:21.275164 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:31:51.274574 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:31:51.274536 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:32:21.275438 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:32:21.275405 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:32:36.916256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:36.916183 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j"] Nov 03 19:32:36.919451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:36.919434 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:36.923688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:36.923652 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 19:32:36.923688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:36.923681 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 19:32:36.925137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:36.925115 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-lmf64\"" Nov 03 19:32:36.928267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:36.928215 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j"] Nov 03 19:32:37.026204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.026171 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.026204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.026205 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2sxk\" (UniqueName: \"kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.026414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.026308 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.127292 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.127257 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.127292 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.127295 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2sxk\" (UniqueName: \"kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.127525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.127366 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.127649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.127631 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.127722 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.127706 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.140503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.140477 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2sxk\" (UniqueName: \"kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.239729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.239706 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:32:37.511690 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.511618 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j"] Nov 03 19:32:37.515059 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:37.515033 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac20584b_144d_4199_9f5e_d66021fd3a6e.slice/crio-1e7deaf5e019fb4ab2dbc126c441331e0bdc9444c91f329a15ddbc8081fee7c8 WatchSource:0}: Error finding container 1e7deaf5e019fb4ab2dbc126c441331e0bdc9444c91f329a15ddbc8081fee7c8: Status 404 returned error can't find the container with id 1e7deaf5e019fb4ab2dbc126c441331e0bdc9444c91f329a15ddbc8081fee7c8 Nov 03 19:32:37.517396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:37.517379 5233 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:32:38.013194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:38.013157 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" event={"ID":"ac20584b-144d-4199-9f5e-d66021fd3a6e","Type":"ContainerStarted","Data":"1e7deaf5e019fb4ab2dbc126c441331e0bdc9444c91f329a15ddbc8081fee7c8"} Nov 03 19:32:40.000128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.000094 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw"] Nov 03 19:32:40.003605 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.003580 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.006238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.006211 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz"] Nov 03 19:32:40.009535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.009499 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 19:32:40.009655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.009535 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:32:40.009815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.009798 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.011597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.011578 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:32:40.015623 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.015603 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ndffd\"" Nov 03 19:32:40.017535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.017501 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-8hzbf\"" Nov 03 19:32:40.020583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.020564 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw"] Nov 03 19:32:40.021229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.021208 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz"] Nov 03 19:32:40.082818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.082782 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf"] Nov 03 19:32:40.088077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.087519 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.102544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.102519 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf"] Nov 03 19:32:40.103913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.103891 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m"] Nov 03 19:32:40.107352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.107332 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.115448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.115419 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-tb7k4\"" Nov 03 19:32:40.115545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.115471 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xqx5c\"" Nov 03 19:32:40.118795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.118765 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m"] Nov 03 19:32:40.150747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.150714 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8bd\" (UniqueName: \"kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.150882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.150762 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.150882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.150859 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.150998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.150931 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.150998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.150963 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.151099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151004 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.151099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151070 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.151205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151104 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.151205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151140 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.151205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151170 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.151325 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151212 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.151325 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.151236 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r45q\" (UniqueName: \"kubernetes.io/projected/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-kube-api-access-8r45q\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.190704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.190675 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d"] Nov 03 19:32:40.194580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.194559 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.202329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.202310 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d"] Nov 03 19:32:40.206921 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.206902 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-zb8p5\"" Nov 03 19:32:40.252085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.251998 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252053 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r45q\" (UniqueName: \"kubernetes.io/projected/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-kube-api-access-8r45q\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252157 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.252282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252199 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcsm\" (UniqueName: \"kubernetes.io/projected/f60ce991-dc22-432c-94f5-dd0f6b184c9c-kube-api-access-dkcsm\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.252282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252236 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8bd\" (UniqueName: \"kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.252448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252310 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252601 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252664 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252694 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252726 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.252913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252765 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.252913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.252798 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.253150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253033 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.253150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253057 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.253307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253283 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.253627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253580 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.253714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253638 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.253714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253667 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmlg\" (UniqueName: \"kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.253853 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.253730 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.254097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254057 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.254097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254077 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.254261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254099 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.254261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254148 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.254404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254385 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.254404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254391 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.254519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254440 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.254519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254473 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.254519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254497 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.254671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254530 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.254671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254621 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.254904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.254879 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.255264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.255241 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.255375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.255247 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.264796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.264765 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r45q\" (UniqueName: \"kubernetes.io/projected/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-kube-api-access-8r45q\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.265110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.265088 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8bd\" (UniqueName: \"kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.266463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.266438 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6929ca2-35e3-40c0-8010-49b9dc0a68c5-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-8tfdw\" (UID: \"d6929ca2-35e3-40c0-8010-49b9dc0a68c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.294507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.294483 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt"] Nov 03 19:32:40.299363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.299347 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.306008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.305962 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lg8bn\"" Nov 03 19:32:40.308218 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.308196 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt"] Nov 03 19:32:40.321747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.321726 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:40.334543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.334523 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355311 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355347 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355373 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355409 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcsm\" (UniqueName: \"kubernetes.io/projected/f60ce991-dc22-432c-94f5-dd0f6b184c9c-kube-api-access-dkcsm\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355454 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355471 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355494 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355511 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355529 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355544 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmlg\" (UniqueName: \"kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355578 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355598 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355613 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355634 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrgd\" (UniqueName: \"kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.355657 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.356608 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.356804 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.357039 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.358067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.357473 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.359768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.359082 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.359768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.359325 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.359768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.359554 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.360174 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.360123 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.361000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.360756 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.369471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.369423 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60ce991-dc22-432c-94f5-dd0f6b184c9c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.370294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.370270 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcsm\" (UniqueName: \"kubernetes.io/projected/f60ce991-dc22-432c-94f5-dd0f6b184c9c-kube-api-access-dkcsm\") pod \"openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf\" (UID: \"f60ce991-dc22-432c-94f5-dd0f6b184c9c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.371456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.371416 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmlg\" (UniqueName: \"kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.408113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.407606 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:40.428203 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.425753 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:32:40.456586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.456553 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.456730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.456616 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.456730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.456667 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.456730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.456702 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrgd\" (UniqueName: \"kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.456863 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.456740 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc62x\" (UniqueName: \"kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.456863 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.456801 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.457281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.457259 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.457528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.457509 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.470381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.470351 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrgd\" (UniqueName: \"kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.512338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.512309 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:32:40.558254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.558220 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc62x\" (UniqueName: \"kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.558388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.558290 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.558388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.558349 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.558765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.558739 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.558855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.558826 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.572510 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.572482 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc62x\" (UniqueName: \"kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.614790 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.614751 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw"] Nov 03 19:32:40.617914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.617894 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:32:40.618223 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:40.618192 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6929ca2_35e3_40c0_8010_49b9dc0a68c5.slice/crio-6b88beef8ee90bca3afd1161b1bc9147a1fd2f4ba2c0e17e1f658ab58a4a3668 WatchSource:0}: Error finding container 6b88beef8ee90bca3afd1161b1bc9147a1fd2f4ba2c0e17e1f658ab58a4a3668: Status 404 returned error can't find the container with id 6b88beef8ee90bca3afd1161b1bc9147a1fd2f4ba2c0e17e1f658ab58a4a3668 Nov 03 19:32:40.643514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.643481 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz"] Nov 03 19:32:40.648113 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:40.648076 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe697380_38ca_470a_af6c_73bc1e2792f3.slice/crio-13b4cae40891647c22b69d81bd8676b71b2a3de4c8fea9c9725d206408061b8f WatchSource:0}: Error finding container 13b4cae40891647c22b69d81bd8676b71b2a3de4c8fea9c9725d206408061b8f: Status 404 returned error can't find the container with id 13b4cae40891647c22b69d81bd8676b71b2a3de4c8fea9c9725d206408061b8f Nov 03 19:32:40.707788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.707758 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf"] Nov 03 19:32:40.710867 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:40.710832 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60ce991_dc22_432c_94f5_dd0f6b184c9c.slice/crio-0af3913eacc8b03dbfabdcd0c58b122a4fe8667c1ee6383839e1e9a20bf237eb WatchSource:0}: Error finding container 0af3913eacc8b03dbfabdcd0c58b122a4fe8667c1ee6383839e1e9a20bf237eb: Status 404 returned error can't find the container with id 0af3913eacc8b03dbfabdcd0c58b122a4fe8667c1ee6383839e1e9a20bf237eb Nov 03 19:32:40.715274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.715249 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m"] Nov 03 19:32:40.718758 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:40.718732 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b1ab57_e117_43d3_b415_8a3d42f742f2.slice/crio-7cdc7c640948d2f13d89b65b99aec584bcce19606020b4056cb8798d506446ca WatchSource:0}: Error finding container 7cdc7c640948d2f13d89b65b99aec584bcce19606020b4056cb8798d506446ca: Status 404 returned error can't find the container with id 7cdc7c640948d2f13d89b65b99aec584bcce19606020b4056cb8798d506446ca Nov 03 19:32:40.794684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.794620 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d"] Nov 03 19:32:40.798086 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:40.798058 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75311111_6ba8_4814_b7ca_1d75519a66c4.slice/crio-afd4d452b6abbdf50fa00c443084994b86313d2242d6dd2e39aa4550e6a1738a WatchSource:0}: Error finding container afd4d452b6abbdf50fa00c443084994b86313d2242d6dd2e39aa4550e6a1738a: Status 404 returned error can't find the container with id afd4d452b6abbdf50fa00c443084994b86313d2242d6dd2e39aa4550e6a1738a Nov 03 19:32:40.899674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:40.899648 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt"] Nov 03 19:32:40.903278 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:40.903253 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311df254_ab9f_4d45_9164_acca7fa1c7a7.slice/crio-76754651fc88987f3240db74002c25890584ebc0c50287d6dd4ea14244fb9103 WatchSource:0}: Error finding container 76754651fc88987f3240db74002c25890584ebc0c50287d6dd4ea14244fb9103: Status 404 returned error can't find the container with id 76754651fc88987f3240db74002c25890584ebc0c50287d6dd4ea14244fb9103 Nov 03 19:32:41.027335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:41.027297 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" event={"ID":"75311111-6ba8-4814-b7ca-1d75519a66c4","Type":"ContainerStarted","Data":"afd4d452b6abbdf50fa00c443084994b86313d2242d6dd2e39aa4550e6a1738a"} Nov 03 19:32:41.028451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:41.028425 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" event={"ID":"33b1ab57-e117-43d3-b415-8a3d42f742f2","Type":"ContainerStarted","Data":"7cdc7c640948d2f13d89b65b99aec584bcce19606020b4056cb8798d506446ca"} Nov 03 19:32:41.029493 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:41.029472 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" event={"ID":"be697380-38ca-470a-af6c-73bc1e2792f3","Type":"ContainerStarted","Data":"13b4cae40891647c22b69d81bd8676b71b2a3de4c8fea9c9725d206408061b8f"} Nov 03 19:32:41.030801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:41.030767 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" event={"ID":"311df254-ab9f-4d45-9164-acca7fa1c7a7","Type":"ContainerStarted","Data":"76754651fc88987f3240db74002c25890584ebc0c50287d6dd4ea14244fb9103"} Nov 03 19:32:41.032153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:41.032130 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" event={"ID":"f60ce991-dc22-432c-94f5-dd0f6b184c9c","Type":"ContainerStarted","Data":"0af3913eacc8b03dbfabdcd0c58b122a4fe8667c1ee6383839e1e9a20bf237eb"} Nov 03 19:32:41.033524 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:41.033503 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" event={"ID":"d6929ca2-35e3-40c0-8010-49b9dc0a68c5","Type":"ContainerStarted","Data":"6b88beef8ee90bca3afd1161b1bc9147a1fd2f4ba2c0e17e1f658ab58a4a3668"} Nov 03 19:32:47.258780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:47.258728 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:32:47.259238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:47.258829 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:32:47.259238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:47.258898 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:32:47.511422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:47.511263 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:32:47.511422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:47.511350 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:32:47.511422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:47.511398 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:32:48.094109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.092234 5233 generic.go:356] "Generic (PLEG): container finished" podID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerID="c019ab662246d49aded9971eea6695b846c8af47ea9a40b366906b07eb335d38" exitCode=0 Nov 03 19:32:48.094109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.093732 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" event={"ID":"ac20584b-144d-4199-9f5e-d66021fd3a6e","Type":"ContainerDied","Data":"c019ab662246d49aded9971eea6695b846c8af47ea9a40b366906b07eb335d38"} Nov 03 19:32:48.097780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.097749 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" event={"ID":"f60ce991-dc22-432c-94f5-dd0f6b184c9c","Type":"ContainerStarted","Data":"9882da0d097bb5b6526b29332dd0bf379f1046f0a4e793460beb32497005e41e"} Nov 03 19:32:48.104121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.104098 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" event={"ID":"d6929ca2-35e3-40c0-8010-49b9dc0a68c5","Type":"ContainerStarted","Data":"877215498c1d73080faae0c4f4a2fbcf5f15415a30a00f09bee1aa1525047617"} Nov 03 19:32:48.140679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.140424 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" podStartSLOduration=1.5961367 podStartE2EDuration="8.140408089s" totalImagesPullingTime="6.544271389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:48.139820859 +0000 UTC m=+334.480934344" watchObservedRunningTime="2026-11-03 19:32:48.140408089 +0000 UTC m=+334.481521572" Nov 03 19:32:48.166751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.166648 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podStartSLOduration=2.2764512679999998 podStartE2EDuration="9.166630072s" totalImagesPullingTime="6.890178804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:48.165315707 +0000 UTC m=+334.506429193" watchObservedRunningTime="2026-11-03 19:32:48.166630072 +0000 UTC m=+334.507743558" Nov 03 19:32:48.322769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.322732 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:48.329105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.329075 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:48.408372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.408344 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:48.413227 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:48.413206 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:49.110274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.110236 5233 generic.go:356] "Generic (PLEG): container finished" podID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerID="cafa5c74bdc99518a74f53a08b756d04b1fdf3eab7b4e89bd48e6a1cfca53d99" exitCode=0 Nov 03 19:32:49.110428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.110320 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" event={"ID":"ac20584b-144d-4199-9f5e-d66021fd3a6e","Type":"ContainerDied","Data":"cafa5c74bdc99518a74f53a08b756d04b1fdf3eab7b4e89bd48e6a1cfca53d99"} Nov 03 19:32:49.111878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.111856 5233 generic.go:356] "Generic (PLEG): container finished" podID="be697380-38ca-470a-af6c-73bc1e2792f3" containerID="7c0513dbc7615741301dd6bfa13097553fe278b7a2f4b0cfd60f4dc0f2fa7a48" exitCode=0 Nov 03 19:32:49.112008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.111929 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" event={"ID":"be697380-38ca-470a-af6c-73bc1e2792f3","Type":"ContainerDied","Data":"7c0513dbc7615741301dd6bfa13097553fe278b7a2f4b0cfd60f4dc0f2fa7a48"} Nov 03 19:32:49.113567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.113541 5233 generic.go:356] "Generic (PLEG): container finished" podID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerID="292a248b448ef1d17b9a973e347867e51d720300ab1345f1a08ad00d26a85bd9" exitCode=0 Nov 03 19:32:49.113684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.113628 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" event={"ID":"311df254-ab9f-4d45-9164-acca7fa1c7a7","Type":"ContainerDied","Data":"292a248b448ef1d17b9a973e347867e51d720300ab1345f1a08ad00d26a85bd9"} Nov 03 19:32:49.115164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.115127 5233 generic.go:356] "Generic (PLEG): container finished" podID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerID="e3f3f0a511fd25a0f004234efd88e90c68c0b36f8371e8c679a411ad5e555fa8" exitCode=0 Nov 03 19:32:49.115279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.115175 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" event={"ID":"75311111-6ba8-4814-b7ca-1d75519a66c4","Type":"ContainerDied","Data":"e3f3f0a511fd25a0f004234efd88e90c68c0b36f8371e8c679a411ad5e555fa8"} Nov 03 19:32:49.116689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.116668 5233 generic.go:356] "Generic (PLEG): container finished" podID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerID="fd48a1c884e4da4edce6683b1f075aeb4112c387dbbc550f5d24935faf35d61a" exitCode=0 Nov 03 19:32:49.116785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.116736 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" event={"ID":"33b1ab57-e117-43d3-b415-8a3d42f742f2","Type":"ContainerDied","Data":"fd48a1c884e4da4edce6683b1f075aeb4112c387dbbc550f5d24935faf35d61a"} Nov 03 19:32:49.117169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.117149 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:49.117417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.117204 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:49.118351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.118261 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf" Nov 03 19:32:49.118351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:49.118310 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:32:50.126033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:50.125948 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" event={"ID":"75311111-6ba8-4814-b7ca-1d75519a66c4","Type":"ContainerStarted","Data":"c20c6b58f53ef2b8fac235c10c56838f87c4d70e6f7552007a2b133fcc9377d4"} Nov 03 19:32:51.133704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.133666 5233 generic.go:356] "Generic (PLEG): container finished" podID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerID="6d02423f84350046731da67674274cd3712caa0a1455cf63727cb56504efba04" exitCode=0 Nov 03 19:32:51.134179 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.133760 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" event={"ID":"311df254-ab9f-4d45-9164-acca7fa1c7a7","Type":"ContainerDied","Data":"6d02423f84350046731da67674274cd3712caa0a1455cf63727cb56504efba04"} Nov 03 19:32:51.136329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.136306 5233 generic.go:356] "Generic (PLEG): container finished" podID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerID="c20c6b58f53ef2b8fac235c10c56838f87c4d70e6f7552007a2b133fcc9377d4" exitCode=0 Nov 03 19:32:51.136556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.136536 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" event={"ID":"75311111-6ba8-4814-b7ca-1d75519a66c4","Type":"ContainerDied","Data":"c20c6b58f53ef2b8fac235c10c56838f87c4d70e6f7552007a2b133fcc9377d4"} Nov 03 19:32:51.144211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.144182 5233 generic.go:356] "Generic (PLEG): container finished" podID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerID="22891e6b488191f4cfb29f623ad92f83bbfd195a7d1b84ff06c713a55b9b2e5f" exitCode=0 Nov 03 19:32:51.144320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.144301 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" event={"ID":"33b1ab57-e117-43d3-b415-8a3d42f742f2","Type":"ContainerDied","Data":"22891e6b488191f4cfb29f623ad92f83bbfd195a7d1b84ff06c713a55b9b2e5f"} Nov 03 19:32:51.146963 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.146939 5233 generic.go:356] "Generic (PLEG): container finished" podID="be697380-38ca-470a-af6c-73bc1e2792f3" containerID="395c4241d36c96b39b83f77859f43cbc399ffa3d6ef4e5cbe20f6c3a4388eecd" exitCode=0 Nov 03 19:32:51.147238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:51.147175 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" event={"ID":"be697380-38ca-470a-af6c-73bc1e2792f3","Type":"ContainerDied","Data":"395c4241d36c96b39b83f77859f43cbc399ffa3d6ef4e5cbe20f6c3a4388eecd"} Nov 03 19:32:51.280908 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:32:51.280871 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:32:52.540543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.540511 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz"] Nov 03 19:32:52.545749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.545724 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.552201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.552175 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-nf9dg\"" Nov 03 19:32:52.554084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.554044 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz"] Nov 03 19:32:52.585551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.585489 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.585677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.585602 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wpw\" (UniqueName: \"kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.585677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.585668 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.687079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.687038 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.687245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.687111 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.687245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.687168 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wpw\" (UniqueName: \"kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.687923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.687903 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.688360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.688322 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.703844 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.703796 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wpw\" (UniqueName: \"kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:52.871166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:52.871088 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:32:53.185773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:53.183396 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz"] Nov 03 19:32:54.173044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:54.172756 5233 generic.go:356] "Generic (PLEG): container finished" podID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerID="678b2598d317dea4328e3264c9debf73981fbf22613ab7e13dbd377000180b2f" exitCode=0 Nov 03 19:32:54.173044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:54.172876 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" event={"ID":"b4fb7624-43c9-4300-9606-6eef53e01cb1","Type":"ContainerDied","Data":"678b2598d317dea4328e3264c9debf73981fbf22613ab7e13dbd377000180b2f"} Nov 03 19:32:54.173044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:54.172925 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" event={"ID":"b4fb7624-43c9-4300-9606-6eef53e01cb1","Type":"ContainerStarted","Data":"ef07e5d051b0a658038c85d308a57e320a9cd2051f5d2a37ded3c38b306597e3"} Nov 03 19:32:56.953741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:56.953705 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n"] Nov 03 19:32:56.964114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:56.964089 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:56.966273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:56.966159 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n"] Nov 03 19:32:56.968878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:56.968830 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-9xb7x\"" Nov 03 19:32:57.030112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.029900 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpslg\" (UniqueName: \"kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.030112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.030058 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.030349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.030157 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.131509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.130935 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.131509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.131003 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.131509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.131064 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpslg\" (UniqueName: \"kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.131509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.131359 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.131509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.131370 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.149925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.149881 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpslg\" (UniqueName: \"kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.293621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.293585 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:32:57.611440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:57.611364 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n"] Nov 03 19:32:57.623907 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:32:57.623871 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a6a96a_156b_4500_9bd9_c21ca6643ca2.slice/crio-55d244987c87a3431c9dd237c19c9ef7190154b223a40fe5d6dd52ee62b9eff7 WatchSource:0}: Error finding container 55d244987c87a3431c9dd237c19c9ef7190154b223a40fe5d6dd52ee62b9eff7: Status 404 returned error can't find the container with id 55d244987c87a3431c9dd237c19c9ef7190154b223a40fe5d6dd52ee62b9eff7 Nov 03 19:32:58.210001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:58.209357 5233 generic.go:356] "Generic (PLEG): container finished" podID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerID="34aac8811bc15104ae45507042cc711ebf0cf91656f0599c626249ace4e96f2d" exitCode=0 Nov 03 19:32:58.210001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:58.209485 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" event={"ID":"44a6a96a-156b-4500-9bd9-c21ca6643ca2","Type":"ContainerDied","Data":"34aac8811bc15104ae45507042cc711ebf0cf91656f0599c626249ace4e96f2d"} Nov 03 19:32:58.210001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:58.209567 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" event={"ID":"44a6a96a-156b-4500-9bd9-c21ca6643ca2","Type":"ContainerStarted","Data":"55d244987c87a3431c9dd237c19c9ef7190154b223a40fe5d6dd52ee62b9eff7"} Nov 03 19:32:59.219101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:59.219065 5233 generic.go:356] "Generic (PLEG): container finished" podID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerID="6c21dc257e9b28d4ef1a83c4af096448b06eda8abe8e0ad582bc1a56343afd20" exitCode=0 Nov 03 19:32:59.219551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:32:59.219109 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" event={"ID":"b4fb7624-43c9-4300-9606-6eef53e01cb1","Type":"ContainerDied","Data":"6c21dc257e9b28d4ef1a83c4af096448b06eda8abe8e0ad582bc1a56343afd20"} Nov 03 19:33:00.228844 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:00.228813 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" event={"ID":"b4fb7624-43c9-4300-9606-6eef53e01cb1","Type":"ContainerStarted","Data":"d0df43f5d5bf584a4715822d8124b2e203fa39c27604c5c46806d1c9dbb275e1"} Nov 03 19:33:00.236669 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:00.235916 5233 generic.go:356] "Generic (PLEG): container finished" podID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerID="0a1f5311fce729918d9c00070ffc1b608a7230c9798fa42ecaa2bbef10248d45" exitCode=0 Nov 03 19:33:00.236669 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:00.236131 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" event={"ID":"ac20584b-144d-4199-9f5e-d66021fd3a6e","Type":"ContainerDied","Data":"0a1f5311fce729918d9c00070ffc1b608a7230c9798fa42ecaa2bbef10248d45"} Nov 03 19:33:00.317215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:00.317154 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" podStartSLOduration=3.094566555 podStartE2EDuration="8.31713374s" totalImagesPullingTime="5.222567185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:00.290074229 +0000 UTC m=+346.631187715" watchObservedRunningTime="2026-11-03 19:33:00.31713374 +0000 UTC m=+346.658247233" Nov 03 19:33:01.245402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.245299 5233 generic.go:356] "Generic (PLEG): container finished" podID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerID="d0df43f5d5bf584a4715822d8124b2e203fa39c27604c5c46806d1c9dbb275e1" exitCode=0 Nov 03 19:33:01.246750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.246681 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" event={"ID":"b4fb7624-43c9-4300-9606-6eef53e01cb1","Type":"ContainerDied","Data":"d0df43f5d5bf584a4715822d8124b2e203fa39c27604c5c46806d1c9dbb275e1"} Nov 03 19:33:01.494993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.494938 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:33:01.583188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.583110 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util\") pod \"ac20584b-144d-4199-9f5e-d66021fd3a6e\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " Nov 03 19:33:01.583355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.583240 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk\" (UniqueName: \"kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk\") pod \"ac20584b-144d-4199-9f5e-d66021fd3a6e\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " Nov 03 19:33:01.583355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.583298 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle\") pod \"ac20584b-144d-4199-9f5e-d66021fd3a6e\" (UID: \"ac20584b-144d-4199-9f5e-d66021fd3a6e\") " Nov 03 19:33:01.584889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.584845 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle" pod "ac20584b-144d-4199-9f5e-d66021fd3a6e" (UID: "ac20584b-144d-4199-9f5e-d66021fd3a6e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:01.590482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.590437 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk" pod "ac20584b-144d-4199-9f5e-d66021fd3a6e" (UID: "ac20584b-144d-4199-9f5e-d66021fd3a6e"). InnerVolumeSpecName "kube-api-access-g2sxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:01.591552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.591482 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util" pod "ac20584b-144d-4199-9f5e-d66021fd3a6e" (UID: "ac20584b-144d-4199-9f5e-d66021fd3a6e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:01.686134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.686042 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2sxk\" (UniqueName: \"kubernetes.io/projected/ac20584b-144d-4199-9f5e-d66021fd3a6e-kube-api-access-g2sxk\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:01.686134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.686085 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:01.686134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:01.686101 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac20584b-144d-4199-9f5e-d66021fd3a6e-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:02.255835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.255806 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" Nov 03 19:33:02.255835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.255798 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714947j" event={"ID":"ac20584b-144d-4199-9f5e-d66021fd3a6e","Type":"ContainerDied","Data":"1e7deaf5e019fb4ab2dbc126c441331e0bdc9444c91f329a15ddbc8081fee7c8"} Nov 03 19:33:02.256298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.255866 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7deaf5e019fb4ab2dbc126c441331e0bdc9444c91f329a15ddbc8081fee7c8" Nov 03 19:33:02.477355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.477319 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:33:02.607430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.607356 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw\" (UniqueName: \"kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw\") pod \"b4fb7624-43c9-4300-9606-6eef53e01cb1\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " Nov 03 19:33:02.607591 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.607451 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util\") pod \"b4fb7624-43c9-4300-9606-6eef53e01cb1\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " Nov 03 19:33:02.607591 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.607471 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle\") pod \"b4fb7624-43c9-4300-9606-6eef53e01cb1\" (UID: \"b4fb7624-43c9-4300-9606-6eef53e01cb1\") " Nov 03 19:33:02.608730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.608700 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle" pod "b4fb7624-43c9-4300-9606-6eef53e01cb1" (UID: "b4fb7624-43c9-4300-9606-6eef53e01cb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:02.614128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.614094 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util" pod "b4fb7624-43c9-4300-9606-6eef53e01cb1" (UID: "b4fb7624-43c9-4300-9606-6eef53e01cb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:02.614756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.614723 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw" pod "b4fb7624-43c9-4300-9606-6eef53e01cb1" (UID: "b4fb7624-43c9-4300-9606-6eef53e01cb1"). InnerVolumeSpecName "kube-api-access-d8wpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:02.708762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.708665 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:02.708762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.708707 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wpw\" (UniqueName: \"kubernetes.io/projected/b4fb7624-43c9-4300-9606-6eef53e01cb1-kube-api-access-d8wpw\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:02.708762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:02.708721 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4fb7624-43c9-4300-9606-6eef53e01cb1-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:03.263214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:03.263150 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" event={"ID":"b4fb7624-43c9-4300-9606-6eef53e01cb1","Type":"ContainerDied","Data":"ef07e5d051b0a658038c85d308a57e320a9cd2051f5d2a37ded3c38b306597e3"} Nov 03 19:33:03.263214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:03.263204 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef07e5d051b0a658038c85d308a57e320a9cd2051f5d2a37ded3c38b306597e3" Nov 03 19:33:03.263778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:03.263277 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crwngz" Nov 03 19:33:04.269498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.269480 5233 generic.go:356] "Generic (PLEG): container finished" podID="be697380-38ca-470a-af6c-73bc1e2792f3" containerID="de5efa8355c3fc71adf3f1bb9760abe82e52e7d0b8d51f67443d83faa1560f4d" exitCode=0 Nov 03 19:33:04.269876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.269563 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" event={"ID":"be697380-38ca-470a-af6c-73bc1e2792f3","Type":"ContainerDied","Data":"de5efa8355c3fc71adf3f1bb9760abe82e52e7d0b8d51f67443d83faa1560f4d"} Nov 03 19:33:04.272130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.272103 5233 generic.go:356] "Generic (PLEG): container finished" podID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerID="4fa5b894f3882e8b51298e8887f4b6b43fa1eb360d4d459ac588cafb73ff9039" exitCode=0 Nov 03 19:33:04.272378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.272181 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" event={"ID":"311df254-ab9f-4d45-9164-acca7fa1c7a7","Type":"ContainerDied","Data":"4fa5b894f3882e8b51298e8887f4b6b43fa1eb360d4d459ac588cafb73ff9039"} Nov 03 19:33:04.274752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.274730 5233 generic.go:356] "Generic (PLEG): container finished" podID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerID="745aeb5398bfea0199356bb0f070e9e9d1f1d4c79e6bb12dc4bd369dee7f042d" exitCode=0 Nov 03 19:33:04.274870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.274784 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" event={"ID":"75311111-6ba8-4814-b7ca-1d75519a66c4","Type":"ContainerDied","Data":"745aeb5398bfea0199356bb0f070e9e9d1f1d4c79e6bb12dc4bd369dee7f042d"} Nov 03 19:33:04.632013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.631912 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww"] Nov 03 19:33:04.632716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632691 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="pull" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632719 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="pull" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632758 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="pull" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632765 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="pull" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632779 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="extract" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632787 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="extract" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632808 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="util" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632815 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="util" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632831 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="util" Nov 03 19:33:04.632846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632838 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="util" Nov 03 19:33:04.633224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632869 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="extract" Nov 03 19:33:04.633224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.632876 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="extract" Nov 03 19:33:04.633224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.633039 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac20584b-144d-4199-9f5e-d66021fd3a6e" containerName="extract" Nov 03 19:33:04.633224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.633065 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fb7624-43c9-4300-9606-6eef53e01cb1" containerName="extract" Nov 03 19:33:04.656526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.656498 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.665933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.665871 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 19:33:04.666102 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.666019 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww"] Nov 03 19:33:04.666199 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.666169 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 19:33:04.666403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.666381 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 19:33:04.666471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.666406 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4m7l6\"" Nov 03 19:33:04.667401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.667382 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 19:33:04.667401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.667394 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 19:33:04.827440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.827409 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1eb6b218-6ded-42a8-9667-3bec01071e87-manifests\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.827598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.827452 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-metrics-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.827598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.827476 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.827598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.827540 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98wz\" (UniqueName: \"kubernetes.io/projected/1eb6b218-6ded-42a8-9667-3bec01071e87-kube-api-access-h98wz\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.827699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.827594 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb6b218-6ded-42a8-9667-3bec01071e87-tmp\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.827699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.827621 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-webhook-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.928761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.928673 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1eb6b218-6ded-42a8-9667-3bec01071e87-manifests\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.928761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.928730 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-metrics-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.929003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.928845 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.929003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.928881 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98wz\" (UniqueName: \"kubernetes.io/projected/1eb6b218-6ded-42a8-9667-3bec01071e87-kube-api-access-h98wz\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.929003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.928931 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb6b218-6ded-42a8-9667-3bec01071e87-tmp\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.929172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.929027 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-webhook-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.929172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.929127 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1eb6b218-6ded-42a8-9667-3bec01071e87-manifests\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.931386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.931350 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb6b218-6ded-42a8-9667-3bec01071e87-tmp\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.931680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.931657 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-metrics-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.931993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.931949 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.932101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.932084 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb6b218-6ded-42a8-9667-3bec01071e87-webhook-cert\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:04.941788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:04.941763 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98wz\" (UniqueName: \"kubernetes.io/projected/1eb6b218-6ded-42a8-9667-3bec01071e87-kube-api-access-h98wz\") pod \"opendatahub-operator-controller-manager-7646b49899-f98ww\" (UID: \"1eb6b218-6ded-42a8-9667-3bec01071e87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:05.007350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.007322 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:05.285472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.285436 5233 generic.go:356] "Generic (PLEG): container finished" podID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerID="a5c79ec9d4018a16ff7d34763799e133295ae0ab314427ec2671cdac92269716" exitCode=0 Nov 03 19:33:05.285875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.285514 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" event={"ID":"33b1ab57-e117-43d3-b415-8a3d42f742f2","Type":"ContainerDied","Data":"a5c79ec9d4018a16ff7d34763799e133295ae0ab314427ec2671cdac92269716"} Nov 03 19:33:05.287334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.287312 5233 generic.go:356] "Generic (PLEG): container finished" podID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerID="e5e63655138933ea47b0732a6210151df56dc7f20ac1c2f709e29f5899c2f4c4" exitCode=0 Nov 03 19:33:05.287431 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.287408 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" event={"ID":"44a6a96a-156b-4500-9bd9-c21ca6643ca2","Type":"ContainerDied","Data":"e5e63655138933ea47b0732a6210151df56dc7f20ac1c2f709e29f5899c2f4c4"} Nov 03 19:33:05.320056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.319773 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww"] Nov 03 19:33:05.329265 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:05.329235 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb6b218_6ded_42a8_9667_3bec01071e87.slice/crio-98dbe44954d9760b2b4fca4a8c2fd389b025455593b919659f0d4d605827b503 WatchSource:0}: Error finding container 98dbe44954d9760b2b4fca4a8c2fd389b025455593b919659f0d4d605827b503: Status 404 returned error can't find the container with id 98dbe44954d9760b2b4fca4a8c2fd389b025455593b919659f0d4d605827b503 Nov 03 19:33:05.486848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.486824 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:33:05.512032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.512010 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:33:05.515387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.515368 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:33:05.633755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633685 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle\") pod \"be697380-38ca-470a-af6c-73bc1e2792f3\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " Nov 03 19:33:05.633755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633729 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle\") pod \"75311111-6ba8-4814-b7ca-1d75519a66c4\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " Nov 03 19:33:05.633987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633761 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util\") pod \"be697380-38ca-470a-af6c-73bc1e2792f3\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " Nov 03 19:33:05.633987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633801 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd\" (UniqueName: \"kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd\") pod \"75311111-6ba8-4814-b7ca-1d75519a66c4\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " Nov 03 19:33:05.633987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633831 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x\" (UniqueName: \"kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x\") pod \"311df254-ab9f-4d45-9164-acca7fa1c7a7\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " Nov 03 19:33:05.633987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633923 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util\") pod \"311df254-ab9f-4d45-9164-acca7fa1c7a7\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " Nov 03 19:33:05.633987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.633954 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd\" (UniqueName: \"kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd\") pod \"be697380-38ca-470a-af6c-73bc1e2792f3\" (UID: \"be697380-38ca-470a-af6c-73bc1e2792f3\") " Nov 03 19:33:05.634243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.634016 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle\") pod \"311df254-ab9f-4d45-9164-acca7fa1c7a7\" (UID: \"311df254-ab9f-4d45-9164-acca7fa1c7a7\") " Nov 03 19:33:05.634243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.634052 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util\") pod \"75311111-6ba8-4814-b7ca-1d75519a66c4\" (UID: \"75311111-6ba8-4814-b7ca-1d75519a66c4\") " Nov 03 19:33:05.634400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.634374 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle" pod "be697380-38ca-470a-af6c-73bc1e2792f3" (UID: "be697380-38ca-470a-af6c-73bc1e2792f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:05.634931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.634883 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle" pod "75311111-6ba8-4814-b7ca-1d75519a66c4" (UID: "75311111-6ba8-4814-b7ca-1d75519a66c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:05.635064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.635037 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle" pod "311df254-ab9f-4d45-9164-acca7fa1c7a7" (UID: "311df254-ab9f-4d45-9164-acca7fa1c7a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:05.636415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.636389 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd" pod "75311111-6ba8-4814-b7ca-1d75519a66c4" (UID: "75311111-6ba8-4814-b7ca-1d75519a66c4"). InnerVolumeSpecName "kube-api-access-7wrgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:05.636649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.636632 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd" pod "be697380-38ca-470a-af6c-73bc1e2792f3" (UID: "be697380-38ca-470a-af6c-73bc1e2792f3"). InnerVolumeSpecName "kube-api-access-7b8bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:05.636784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.636767 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x" pod "311df254-ab9f-4d45-9164-acca7fa1c7a7" (UID: "311df254-ab9f-4d45-9164-acca7fa1c7a7"). InnerVolumeSpecName "kube-api-access-qc62x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:05.640370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.640349 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util" pod "be697380-38ca-470a-af6c-73bc1e2792f3" (UID: "be697380-38ca-470a-af6c-73bc1e2792f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:05.642342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.642317 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util" pod "311df254-ab9f-4d45-9164-acca7fa1c7a7" (UID: "311df254-ab9f-4d45-9164-acca7fa1c7a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:05.642792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.642766 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util" pod "75311111-6ba8-4814-b7ca-1d75519a66c4" (UID: "75311111-6ba8-4814-b7ca-1d75519a66c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:05.735263 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735234 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735263 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735261 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8bd\" (UniqueName: \"kubernetes.io/projected/be697380-38ca-470a-af6c-73bc1e2792f3-kube-api-access-7b8bd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735276 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/311df254-ab9f-4d45-9164-acca7fa1c7a7-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735288 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735299 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735311 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75311111-6ba8-4814-b7ca-1d75519a66c4-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735322 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be697380-38ca-470a-af6c-73bc1e2792f3-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735333 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrgd\" (UniqueName: \"kubernetes.io/projected/75311111-6ba8-4814-b7ca-1d75519a66c4-kube-api-access-7wrgd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:05.735425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:05.735346 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc62x\" (UniqueName: \"kubernetes.io/projected/311df254-ab9f-4d45-9164-acca7fa1c7a7-kube-api-access-qc62x\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:06.295801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.295761 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" event={"ID":"1eb6b218-6ded-42a8-9667-3bec01071e87","Type":"ContainerStarted","Data":"98dbe44954d9760b2b4fca4a8c2fd389b025455593b919659f0d4d605827b503"} Nov 03 19:33:06.298666 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.298635 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" event={"ID":"be697380-38ca-470a-af6c-73bc1e2792f3","Type":"ContainerDied","Data":"13b4cae40891647c22b69d81bd8676b71b2a3de4c8fea9c9725d206408061b8f"} Nov 03 19:33:06.298803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.298684 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b4cae40891647c22b69d81bd8676b71b2a3de4c8fea9c9725d206408061b8f" Nov 03 19:33:06.298803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.298763 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qczkz" Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.303160 5233 generic.go:356] "Generic (PLEG): container finished" podID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerID="3bf075d341caa0e550ed6a1389adbec611be67ae90fed3a61b8e89e0c68d0840" exitCode=0 Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.303265 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" event={"ID":"44a6a96a-156b-4500-9bd9-c21ca6643ca2","Type":"ContainerDied","Data":"3bf075d341caa0e550ed6a1389adbec611be67ae90fed3a61b8e89e0c68d0840"} Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.305483 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" event={"ID":"311df254-ab9f-4d45-9164-acca7fa1c7a7","Type":"ContainerDied","Data":"76754651fc88987f3240db74002c25890584ebc0c50287d6dd4ea14244fb9103"} Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.305502 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqcjt" Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.305537 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76754651fc88987f3240db74002c25890584ebc0c50287d6dd4ea14244fb9103" Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.309811 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.310293 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqg86d" event={"ID":"75311111-6ba8-4814-b7ca-1d75519a66c4","Type":"ContainerDied","Data":"afd4d452b6abbdf50fa00c443084994b86313d2242d6dd2e39aa4550e6a1738a"} Nov 03 19:33:06.323011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.310323 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd4d452b6abbdf50fa00c443084994b86313d2242d6dd2e39aa4550e6a1738a" Nov 03 19:33:06.473248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.473218 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:33:06.649554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.649469 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle\") pod \"33b1ab57-e117-43d3-b415-8a3d42f742f2\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " Nov 03 19:33:06.649554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.649552 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util\") pod \"33b1ab57-e117-43d3-b415-8a3d42f742f2\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " Nov 03 19:33:06.649769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.649612 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg\" (UniqueName: \"kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg\") pod \"33b1ab57-e117-43d3-b415-8a3d42f742f2\" (UID: \"33b1ab57-e117-43d3-b415-8a3d42f742f2\") " Nov 03 19:33:06.652101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.652068 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle" pod "33b1ab57-e117-43d3-b415-8a3d42f742f2" (UID: "33b1ab57-e117-43d3-b415-8a3d42f742f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:06.652206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.652122 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg" pod "33b1ab57-e117-43d3-b415-8a3d42f742f2" (UID: "33b1ab57-e117-43d3-b415-8a3d42f742f2"). InnerVolumeSpecName "kube-api-access-pdmlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:06.656140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.656110 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util" pod "33b1ab57-e117-43d3-b415-8a3d42f742f2" (UID: "33b1ab57-e117-43d3-b415-8a3d42f742f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:06.751188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.751148 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:06.751188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.751186 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33b1ab57-e117-43d3-b415-8a3d42f742f2-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:06.751397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:06.751201 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdmlg\" (UniqueName: \"kubernetes.io/projected/33b1ab57-e117-43d3-b415-8a3d42f742f2-kube-api-access-pdmlg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.317376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.317345 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" Nov 03 19:33:07.317376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.317368 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ctd5m" event={"ID":"33b1ab57-e117-43d3-b415-8a3d42f742f2","Type":"ContainerDied","Data":"7cdc7c640948d2f13d89b65b99aec584bcce19606020b4056cb8798d506446ca"} Nov 03 19:33:07.317875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.317424 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cdc7c640948d2f13d89b65b99aec584bcce19606020b4056cb8798d506446ca" Nov 03 19:33:07.458999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.458961 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:33:07.560397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.560357 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle\") pod \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " Nov 03 19:33:07.560555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.560431 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util\") pod \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " Nov 03 19:33:07.560555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.560517 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg\" (UniqueName: \"kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg\") pod \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\" (UID: \"44a6a96a-156b-4500-9bd9-c21ca6643ca2\") " Nov 03 19:33:07.560825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.560782 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle" pod "44a6a96a-156b-4500-9bd9-c21ca6643ca2" (UID: "44a6a96a-156b-4500-9bd9-c21ca6643ca2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:07.564694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.564652 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util" pod "44a6a96a-156b-4500-9bd9-c21ca6643ca2" (UID: "44a6a96a-156b-4500-9bd9-c21ca6643ca2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:07.566214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.566186 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg" pod "44a6a96a-156b-4500-9bd9-c21ca6643ca2" (UID: "44a6a96a-156b-4500-9bd9-c21ca6643ca2"). InnerVolumeSpecName "kube-api-access-xpslg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:07.661291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.661215 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpslg\" (UniqueName: \"kubernetes.io/projected/44a6a96a-156b-4500-9bd9-c21ca6643ca2-kube-api-access-xpslg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.661291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.661248 5233 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-bundle\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:07.661291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:07.661261 5233 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44a6a96a-156b-4500-9bd9-c21ca6643ca2-util\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:08.322868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:08.322834 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" event={"ID":"1eb6b218-6ded-42a8-9667-3bec01071e87","Type":"ContainerStarted","Data":"269734832e6abf32b2c3d91c34d0268f49c8aeebe55f989b408d4c1348cfe9d7"} Nov 03 19:33:08.324765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:08.324737 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" event={"ID":"44a6a96a-156b-4500-9bd9-c21ca6643ca2","Type":"ContainerDied","Data":"55d244987c87a3431c9dd237c19c9ef7190154b223a40fe5d6dd52ee62b9eff7"} Nov 03 19:33:08.324765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:08.324759 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f6c9n" Nov 03 19:33:08.324929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:08.324777 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d244987c87a3431c9dd237c19c9ef7190154b223a40fe5d6dd52ee62b9eff7" Nov 03 19:33:09.330397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:09.330365 5233 generic.go:356] "Generic (PLEG): container finished" podID="1eb6b218-6ded-42a8-9667-3bec01071e87" containerID="269734832e6abf32b2c3d91c34d0268f49c8aeebe55f989b408d4c1348cfe9d7" exitCode=0 Nov 03 19:33:09.330789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:09.330451 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" event={"ID":"1eb6b218-6ded-42a8-9667-3bec01071e87","Type":"ContainerDied","Data":"269734832e6abf32b2c3d91c34d0268f49c8aeebe55f989b408d4c1348cfe9d7"} Nov 03 19:33:10.336099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:10.336063 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" event={"ID":"1eb6b218-6ded-42a8-9667-3bec01071e87","Type":"ContainerStarted","Data":"58c44b2ea02662e03861162cc91a580271df890fb4fd7911c0209489c53ef629"} Nov 03 19:33:10.336490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:10.336180 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:10.395641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:10.395590 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" podStartSLOduration=2.619900173 podStartE2EDuration="6.395577975s" totalImagesPullingTime="3.360604862s" totalInitContainerRuntime="415.07294ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:10.388830698 +0000 UTC m=+356.729944183" watchObservedRunningTime="2026-11-03 19:33:10.395577975 +0000 UTC m=+356.736691494" Nov 03 19:33:16.532901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.532868 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql"] Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533173 5233 cpu_manager.go:414] "containerMap: removing container" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533184 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533194 5233 cpu_manager.go:414] "containerMap: removing container" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533199 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533208 5233 cpu_manager.go:414] "containerMap: removing container" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="util" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533213 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="util" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533220 5233 cpu_manager.go:414] "containerMap: removing container" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533225 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533233 5233 cpu_manager.go:414] "containerMap: removing container" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533237 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533247 5233 cpu_manager.go:414] "containerMap: removing container" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533253 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="pull" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533258 5233 cpu_manager.go:414] "containerMap: removing container" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="extract" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533262 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="extract" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533270 5233 cpu_manager.go:414] "containerMap: removing container" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="extract" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533275 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="extract" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533281 5233 cpu_manager.go:414] "containerMap: removing container" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="extract" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533286 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="extract" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533295 5233 cpu_manager.go:414] "containerMap: removing container" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="util" Nov 03 19:33:16.533290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533300 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533305 5233 cpu_manager.go:414] "containerMap: removing container" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533310 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533317 5233 cpu_manager.go:414] "containerMap: removing container" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533321 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533330 5233 cpu_manager.go:414] "containerMap: removing container" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533334 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533342 5233 cpu_manager.go:414] "containerMap: removing container" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533346 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="util" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533351 5233 cpu_manager.go:414] "containerMap: removing container" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533355 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533396 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="311df254-ab9f-4d45-9164-acca7fa1c7a7" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533403 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a6a96a-156b-4500-9bd9-c21ca6643ca2" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533411 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="be697380-38ca-470a-af6c-73bc1e2792f3" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533416 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b1ab57-e117-43d3-b415-8a3d42f742f2" containerName="extract" Nov 03 19:33:16.533889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.533421 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="75311111-6ba8-4814-b7ca-1d75519a66c4" containerName="extract" Nov 03 19:33:16.539518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.539500 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:16.547916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.547143 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 19:33:16.547916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.547156 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 19:33:16.548735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.548716 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 19:33:16.548860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.548838 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-smnz4\"" Nov 03 19:33:16.550811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.550792 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql"] Nov 03 19:33:16.631532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.631499 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jllld\" (UniqueName: \"kubernetes.io/projected/b11d7ca3-d9e1-4bed-8705-693f2351fe66-kube-api-access-jllld\") pod \"dns-operator-controller-manager-8557cbdb84-8p5ql\" (UID: \"b11d7ca3-d9e1-4bed-8705-693f2351fe66\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:16.732831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.732802 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jllld\" (UniqueName: \"kubernetes.io/projected/b11d7ca3-d9e1-4bed-8705-693f2351fe66-kube-api-access-jllld\") pod \"dns-operator-controller-manager-8557cbdb84-8p5ql\" (UID: \"b11d7ca3-d9e1-4bed-8705-693f2351fe66\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:16.746820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.746791 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jllld\" (UniqueName: \"kubernetes.io/projected/b11d7ca3-d9e1-4bed-8705-693f2351fe66-kube-api-access-jllld\") pod \"dns-operator-controller-manager-8557cbdb84-8p5ql\" (UID: \"b11d7ca3-d9e1-4bed-8705-693f2351fe66\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:16.852839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:16.852765 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:17.153414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:17.153336 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql"] Nov 03 19:33:17.156469 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:17.156441 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11d7ca3_d9e1_4bed_8705_693f2351fe66.slice/crio-4b1930d279ef375aec19cef94f9dadadc6cd526fb8b3e56a1f3e72bd5c445af8 WatchSource:0}: Error finding container 4b1930d279ef375aec19cef94f9dadadc6cd526fb8b3e56a1f3e72bd5c445af8: Status 404 returned error can't find the container with id 4b1930d279ef375aec19cef94f9dadadc6cd526fb8b3e56a1f3e72bd5c445af8 Nov 03 19:33:17.371954 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:17.371917 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" event={"ID":"b11d7ca3-d9e1-4bed-8705-693f2351fe66","Type":"ContainerStarted","Data":"4b1930d279ef375aec19cef94f9dadadc6cd526fb8b3e56a1f3e72bd5c445af8"} Nov 03 19:33:21.279485 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:33:21.279451 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:33:21.336165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.336129 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw"] Nov 03 19:33:21.344902 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.344872 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:21.346718 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.346695 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7646b49899-f98ww" Nov 03 19:33:21.361582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.361558 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8dv5w\"" Nov 03 19:33:21.371268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.370539 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcwq\" (UniqueName: \"kubernetes.io/projected/b2be9f43-240d-4ded-b132-5e15b27f689b-kube-api-access-4lcwq\") pod \"limitador-operator-controller-manager-5778bb4449-cdmlw\" (UID: \"b2be9f43-240d-4ded-b132-5e15b27f689b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:21.372530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.372506 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw"] Nov 03 19:33:21.473346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.473313 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcwq\" (UniqueName: \"kubernetes.io/projected/b2be9f43-240d-4ded-b132-5e15b27f689b-kube-api-access-4lcwq\") pod \"limitador-operator-controller-manager-5778bb4449-cdmlw\" (UID: \"b2be9f43-240d-4ded-b132-5e15b27f689b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:21.474028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.473956 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9"] Nov 03 19:33:21.484586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.484562 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.490120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.490086 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcwq\" (UniqueName: \"kubernetes.io/projected/b2be9f43-240d-4ded-b132-5e15b27f689b-kube-api-access-4lcwq\") pod \"limitador-operator-controller-manager-5778bb4449-cdmlw\" (UID: \"b2be9f43-240d-4ded-b132-5e15b27f689b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:21.490689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.490627 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:33:21.490864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.490826 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8brlb\"" Nov 03 19:33:21.490864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.490831 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 19:33:21.495632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.495603 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9"] Nov 03 19:33:21.574892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.574816 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74zs\" (UniqueName: \"kubernetes.io/projected/90b6032d-e04a-478c-99f3-16bf24d77e65-kube-api-access-t74zs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9\" (UID: \"90b6032d-e04a-478c-99f3-16bf24d77e65\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.574892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.574858 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b6032d-e04a-478c-99f3-16bf24d77e65-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9\" (UID: \"90b6032d-e04a-478c-99f3-16bf24d77e65\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.660385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.660351 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:21.676055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.676026 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74zs\" (UniqueName: \"kubernetes.io/projected/90b6032d-e04a-478c-99f3-16bf24d77e65-kube-api-access-t74zs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9\" (UID: \"90b6032d-e04a-478c-99f3-16bf24d77e65\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.676185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.676065 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b6032d-e04a-478c-99f3-16bf24d77e65-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9\" (UID: \"90b6032d-e04a-478c-99f3-16bf24d77e65\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.676437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.676420 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b6032d-e04a-478c-99f3-16bf24d77e65-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9\" (UID: \"90b6032d-e04a-478c-99f3-16bf24d77e65\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.696396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.696366 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74zs\" (UniqueName: \"kubernetes.io/projected/90b6032d-e04a-478c-99f3-16bf24d77e65-kube-api-access-t74zs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9\" (UID: \"90b6032d-e04a-478c-99f3-16bf24d77e65\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.809990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.809932 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" Nov 03 19:33:21.953513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:21.953487 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw"] Nov 03 19:33:21.956009 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:21.955956 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2be9f43_240d_4ded_b132_5e15b27f689b.slice/crio-69a9bf9e8ef1a2f24a90ef654b76d91edbc9dd35c932b4baca005cadbb7d0d2e WatchSource:0}: Error finding container 69a9bf9e8ef1a2f24a90ef654b76d91edbc9dd35c932b4baca005cadbb7d0d2e: Status 404 returned error can't find the container with id 69a9bf9e8ef1a2f24a90ef654b76d91edbc9dd35c932b4baca005cadbb7d0d2e Nov 03 19:33:22.152025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:22.151924 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9"] Nov 03 19:33:22.155788 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:22.155756 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b6032d_e04a_478c_99f3_16bf24d77e65.slice/crio-866c357d13696c38ae7effb8391a5b438a8782e94bf29a883ff1ceaa840a6487 WatchSource:0}: Error finding container 866c357d13696c38ae7effb8391a5b438a8782e94bf29a883ff1ceaa840a6487: Status 404 returned error can't find the container with id 866c357d13696c38ae7effb8391a5b438a8782e94bf29a883ff1ceaa840a6487 Nov 03 19:33:22.396504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:22.396475 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" event={"ID":"90b6032d-e04a-478c-99f3-16bf24d77e65","Type":"ContainerStarted","Data":"866c357d13696c38ae7effb8391a5b438a8782e94bf29a883ff1ceaa840a6487"} Nov 03 19:33:22.398003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:22.397953 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" event={"ID":"b2be9f43-240d-4ded-b132-5e15b27f689b","Type":"ContainerStarted","Data":"69a9bf9e8ef1a2f24a90ef654b76d91edbc9dd35c932b4baca005cadbb7d0d2e"} Nov 03 19:33:22.399612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:22.399591 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" event={"ID":"b11d7ca3-d9e1-4bed-8705-693f2351fe66","Type":"ContainerStarted","Data":"20aa4a274d8b8a70b98ae7c707fbc719938727c03a09dc3062ae78bada05a850"} Nov 03 19:33:22.399754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:22.399740 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:22.424582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:22.424513 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" podStartSLOduration=1.304889502 podStartE2EDuration="6.424500963s" totalImagesPullingTime="5.119611461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:22.422937863 +0000 UTC m=+368.764051348" watchObservedRunningTime="2026-11-03 19:33:22.424500963 +0000 UTC m=+368.765614447" Nov 03 19:33:24.412560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:24.412517 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" event={"ID":"b2be9f43-240d-4ded-b132-5e15b27f689b","Type":"ContainerStarted","Data":"468812ac29d2627ff9b0ef9426b9f29b67893ebe5c075bafd91556444081765c"} Nov 03 19:33:24.413002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:24.412644 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:24.446797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:24.446741 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" podStartSLOduration=1.7801664160000001 podStartE2EDuration="3.446727099s" totalImagesPullingTime="1.666560683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:24.445145637 +0000 UTC m=+370.786259123" watchObservedRunningTime="2026-11-03 19:33:24.446727099 +0000 UTC m=+370.787840584" Nov 03 19:33:25.419598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:25.419557 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" event={"ID":"90b6032d-e04a-478c-99f3-16bf24d77e65","Type":"ContainerStarted","Data":"3cdefe362e6eca57ac1978d77b2a4bcd6f25de270e4821137a1f0c98f93175b9"} Nov 03 19:33:25.451450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:25.451392 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-b6kz9" podStartSLOduration=1.667624241 podStartE2EDuration="4.451373452s" totalImagesPullingTime="2.783749211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:25.449234032 +0000 UTC m=+371.790347528" watchObservedRunningTime="2026-11-03 19:33:25.451373452 +0000 UTC m=+371.792486940" Nov 03 19:33:29.862039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:29.862006 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g"] Nov 03 19:33:29.866197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:29.866178 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:29.879581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:29.879561 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g"] Nov 03 19:33:29.888714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:29.888693 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-brtl6\"" Nov 03 19:33:29.950443 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:29.950416 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lcb4g\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:29.950567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:29.950454 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fmq\" (UniqueName: \"kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq\") pod \"kuadrant-operator-controller-manager-678d87766f-lcb4g\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:30.051184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:30.051150 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lcb4g\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:30.051351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:30.051195 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fmq\" (UniqueName: \"kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq\") pod \"kuadrant-operator-controller-manager-678d87766f-lcb4g\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:30.051544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:30.051521 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lcb4g\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:30.074626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:30.074566 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fmq\" (UniqueName: \"kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq\") pod \"kuadrant-operator-controller-manager-678d87766f-lcb4g\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:30.177262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:30.177180 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:30.478783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:30.478750 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g"] Nov 03 19:33:30.483750 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:30.483723 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75643011_dadc_414a_8aeb_0629a61ae5ca.slice/crio-40a216af349b99ff41b706adc356137127c3b47588d14f2cffd5ffb82570706b WatchSource:0}: Error finding container 40a216af349b99ff41b706adc356137127c3b47588d14f2cffd5ffb82570706b: Status 404 returned error can't find the container with id 40a216af349b99ff41b706adc356137127c3b47588d14f2cffd5ffb82570706b Nov 03 19:33:31.450548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:31.450510 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" event={"ID":"75643011-dadc-414a-8aeb-0629a61ae5ca","Type":"ContainerStarted","Data":"40a216af349b99ff41b706adc356137127c3b47588d14f2cffd5ffb82570706b"} Nov 03 19:33:33.408239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:33.408208 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8p5ql" Nov 03 19:33:35.422319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:35.422290 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cdmlw" Nov 03 19:33:36.388899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.388865 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jcfzh"] Nov 03 19:33:36.396186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.396165 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.412107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.412079 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 19:33:36.412217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.412117 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fqmqq\"" Nov 03 19:33:36.412506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.412488 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jcfzh"] Nov 03 19:33:36.416855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.416839 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 19:33:36.508103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.508071 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02195884-11e0-43c5-9aea-72926427bbf1-bound-sa-token\") pod \"cert-manager-597b95cb84-jcfzh\" (UID: \"02195884-11e0-43c5-9aea-72926427bbf1\") " pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.508576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.508154 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dd7p\" (UniqueName: \"kubernetes.io/projected/02195884-11e0-43c5-9aea-72926427bbf1-kube-api-access-8dd7p\") pod \"cert-manager-597b95cb84-jcfzh\" (UID: \"02195884-11e0-43c5-9aea-72926427bbf1\") " pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.609482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.609438 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dd7p\" (UniqueName: \"kubernetes.io/projected/02195884-11e0-43c5-9aea-72926427bbf1-kube-api-access-8dd7p\") pod \"cert-manager-597b95cb84-jcfzh\" (UID: \"02195884-11e0-43c5-9aea-72926427bbf1\") " pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.609691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.609574 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02195884-11e0-43c5-9aea-72926427bbf1-bound-sa-token\") pod \"cert-manager-597b95cb84-jcfzh\" (UID: \"02195884-11e0-43c5-9aea-72926427bbf1\") " pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.638514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.638480 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02195884-11e0-43c5-9aea-72926427bbf1-bound-sa-token\") pod \"cert-manager-597b95cb84-jcfzh\" (UID: \"02195884-11e0-43c5-9aea-72926427bbf1\") " pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.638659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.638559 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dd7p\" (UniqueName: \"kubernetes.io/projected/02195884-11e0-43c5-9aea-72926427bbf1-kube-api-access-8dd7p\") pod \"cert-manager-597b95cb84-jcfzh\" (UID: \"02195884-11e0-43c5-9aea-72926427bbf1\") " pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.719492 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.719463 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jcfzh" Nov 03 19:33:36.993770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.993696 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll"] Nov 03 19:33:36.998372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:36.998352 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.006340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.006321 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fq9v4\"" Nov 03 19:33:37.031370 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:37.031247 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02195884_11e0_43c5_9aea_72926427bbf1.slice/crio-609f17c45257cb39ea7ec6b178dabf9c740a482247bea160f22436a87a500f4b WatchSource:0}: Error finding container 609f17c45257cb39ea7ec6b178dabf9c740a482247bea160f22436a87a500f4b: Status 404 returned error can't find the container with id 609f17c45257cb39ea7ec6b178dabf9c740a482247bea160f22436a87a500f4b Nov 03 19:33:37.034096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.034068 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll"] Nov 03 19:33:37.035517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.035471 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jcfzh"] Nov 03 19:33:37.113595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113555 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5sb\" (UniqueName: \"kubernetes.io/projected/adaf19c0-adea-42f2-84f6-a889dd40eb81-kube-api-access-qb5sb\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113612 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113723 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113753 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-token\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113770 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaf19c0-adea-42f2-84f6-a889dd40eb81-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113851 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-data\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113877 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.113985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.113961 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.114025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.114010 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214364 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-data\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214407 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214430 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214465 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214530 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5sb\" (UniqueName: \"kubernetes.io/projected/adaf19c0-adea-42f2-84f6-a889dd40eb81-kube-api-access-qb5sb\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214564 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214602 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.214641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214623 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-token\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.215068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214650 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaf19c0-adea-42f2-84f6-a889dd40eb81-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.215068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.214890 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.215187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.215070 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.216416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.216389 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-data\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.222079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.217436 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.222079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.217732 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaf19c0-adea-42f2-84f6-a889dd40eb81-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.228258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.228233 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.229365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.229333 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.242519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.242485 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaf19c0-adea-42f2-84f6-a889dd40eb81-istio-token\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.242683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.242659 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5sb\" (UniqueName: \"kubernetes.io/projected/adaf19c0-adea-42f2-84f6-a889dd40eb81-kube-api-access-qb5sb\") pod \"data-science-gateway-data-science-gateway-class-8f76c657-2mnll\" (UID: \"adaf19c0-adea-42f2-84f6-a889dd40eb81\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.311337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.311267 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:37.478530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.478495 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jcfzh" event={"ID":"02195884-11e0-43c5-9aea-72926427bbf1","Type":"ContainerStarted","Data":"609f17c45257cb39ea7ec6b178dabf9c740a482247bea160f22436a87a500f4b"} Nov 03 19:33:37.595261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.595066 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll"] Nov 03 19:33:37.597770 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:37.597745 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaf19c0_adea_42f2_84f6_a889dd40eb81.slice/crio-3fa5484f215072d78a4f82a0cedcb3168bb4f839ad12411bc5a3c6bb61d90eaa WatchSource:0}: Error finding container 3fa5484f215072d78a4f82a0cedcb3168bb4f839ad12411bc5a3c6bb61d90eaa: Status 404 returned error can't find the container with id 3fa5484f215072d78a4f82a0cedcb3168bb4f839ad12411bc5a3c6bb61d90eaa Nov 03 19:33:37.599960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.599927 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:33:37.600073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.600018 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:33:37.600073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:37.600048 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:33:38.487072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:38.487024 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" event={"ID":"adaf19c0-adea-42f2-84f6-a889dd40eb81","Type":"ContainerStarted","Data":"1fb81562d9594615989f3483ea7abc8b44a839c77432dbd8589ce5bba5929ccc"} Nov 03 19:33:38.487273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:38.487086 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" event={"ID":"adaf19c0-adea-42f2-84f6-a889dd40eb81","Type":"ContainerStarted","Data":"3fa5484f215072d78a4f82a0cedcb3168bb4f839ad12411bc5a3c6bb61d90eaa"} Nov 03 19:33:38.549254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:38.549203 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" podStartSLOduration=2.549185579 podStartE2EDuration="2.549185579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:38.548219851 +0000 UTC m=+384.889333338" watchObservedRunningTime="2026-11-03 19:33:38.549185579 +0000 UTC m=+384.890299061" Nov 03 19:33:39.311807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:39.311772 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:39.317920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:39.317846 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:39.495966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:39.495936 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:39.497109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:39.497091 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-8f76c657-2mnll" Nov 03 19:33:40.282193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.282160 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj"] Nov 03 19:33:40.286287 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.286263 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.308056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.308021 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-p48w4\"" Nov 03 19:33:40.312806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.311088 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 19:33:40.312806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.311617 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 19:33:40.312806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.312686 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj"] Nov 03 19:33:40.317044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.315708 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 19:33:40.346726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.346690 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c1dc26e-9699-469c-9432-6006a1210a69-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.346901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.346743 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c1dc26e-9699-469c-9432-6006a1210a69-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.346901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.346895 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c1dc26e-9699-469c-9432-6006a1210a69-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.347083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.346933 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz4b6\" (UniqueName: \"kubernetes.io/projected/7c1dc26e-9699-469c-9432-6006a1210a69-kube-api-access-lz4b6\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.447688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.447654 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c1dc26e-9699-469c-9432-6006a1210a69-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.447862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.447702 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c1dc26e-9699-469c-9432-6006a1210a69-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.447862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.447762 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c1dc26e-9699-469c-9432-6006a1210a69-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.447862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.447804 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz4b6\" (UniqueName: \"kubernetes.io/projected/7c1dc26e-9699-469c-9432-6006a1210a69-kube-api-access-lz4b6\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.448466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.448437 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c1dc26e-9699-469c-9432-6006a1210a69-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.448699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.448679 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c1dc26e-9699-469c-9432-6006a1210a69-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.449173 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.449151 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c1dc26e-9699-469c-9432-6006a1210a69-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.464544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.464518 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz4b6\" (UniqueName: \"kubernetes.io/projected/7c1dc26e-9699-469c-9432-6006a1210a69-kube-api-access-lz4b6\") pod \"kserve-module-controller-manager-7566cf5fc8-mjtbj\" (UID: \"7c1dc26e-9699-469c-9432-6006a1210a69\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.604581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.604554 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:40.899964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:40.899931 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj"] Nov 03 19:33:40.906004 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:40.902791 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1dc26e_9699_469c_9432_6006a1210a69.slice/crio-c50c85fb312af1588d1e131c63fb04e2a1f09cc77d8d032c12eaa6d0d3e0aba1 WatchSource:0}: Error finding container c50c85fb312af1588d1e131c63fb04e2a1f09cc77d8d032c12eaa6d0d3e0aba1: Status 404 returned error can't find the container with id c50c85fb312af1588d1e131c63fb04e2a1f09cc77d8d032c12eaa6d0d3e0aba1 Nov 03 19:33:41.510301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.510261 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" event={"ID":"7c1dc26e-9699-469c-9432-6006a1210a69","Type":"ContainerStarted","Data":"c50c85fb312af1588d1e131c63fb04e2a1f09cc77d8d032c12eaa6d0d3e0aba1"} Nov 03 19:33:41.512455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.512419 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" event={"ID":"75643011-dadc-414a-8aeb-0629a61ae5ca","Type":"ContainerStarted","Data":"3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a"} Nov 03 19:33:41.512614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.512577 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:41.514021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.513997 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jcfzh" event={"ID":"02195884-11e0-43c5-9aea-72926427bbf1","Type":"ContainerStarted","Data":"4a6fb2dffe09e0ec58b6f16d8d73009a7f1c256ba3b9632049d2a9527a4e34e1"} Nov 03 19:33:41.543842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.543499 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" podStartSLOduration=1.981432153 podStartE2EDuration="12.543480874s" totalImagesPullingTime="10.562048721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:41.540820046 +0000 UTC m=+387.881933532" watchObservedRunningTime="2026-11-03 19:33:41.543480874 +0000 UTC m=+387.884594360" Nov 03 19:33:41.570732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.570685 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jcfzh" podStartSLOduration=2.021832211 podStartE2EDuration="5.570671673s" totalImagesPullingTime="3.548839462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:41.569083414 +0000 UTC m=+387.910196899" watchObservedRunningTime="2026-11-03 19:33:41.570671673 +0000 UTC m=+387.911785157" Nov 03 19:33:41.954265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.954189 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:41.969358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.969333 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:41.979930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:41.979863 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:42.008732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.008705 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-frwwl\"" Nov 03 19:33:42.049163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.049136 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 19:33:42.050739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.050714 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:42.063202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.063171 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file\") pod \"limitador-limitador-74dfcfb894-zh594\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.063350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.063262 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgcr\" (UniqueName: \"kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr\") pod \"limitador-limitador-74dfcfb894-zh594\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.163877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.163839 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgcr\" (UniqueName: \"kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr\") pod \"limitador-limitador-74dfcfb894-zh594\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.164088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.163937 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file\") pod \"limitador-limitador-74dfcfb894-zh594\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.164689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.164663 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file\") pod \"limitador-limitador-74dfcfb894-zh594\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.183658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.183630 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgcr\" (UniqueName: \"kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr\") pod \"limitador-limitador-74dfcfb894-zh594\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.314224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.314196 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:42.611704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:42.611536 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:42.614326 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:42.614292 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde3f988_8444_42aa_8d40_64830292a2a3.slice/crio-798779421be398a757d43c3d99df436a05adc83d9be3f0f725e69ef5a7696e56 WatchSource:0}: Error finding container 798779421be398a757d43c3d99df436a05adc83d9be3f0f725e69ef5a7696e56: Status 404 returned error can't find the container with id 798779421be398a757d43c3d99df436a05adc83d9be3f0f725e69ef5a7696e56 Nov 03 19:33:43.525930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:43.525897 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" event={"ID":"cde3f988-8444-42aa-8d40-64830292a2a3","Type":"ContainerStarted","Data":"798779421be398a757d43c3d99df436a05adc83d9be3f0f725e69ef5a7696e56"} Nov 03 19:33:44.532370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:44.532335 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" event={"ID":"7c1dc26e-9699-469c-9432-6006a1210a69","Type":"ContainerStarted","Data":"e19cfe4164107cb2978c21d1d1968512b1526480ec0f931f55999635b7d5475d"} Nov 03 19:33:44.532833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:44.532396 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:33:44.533844 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:44.533817 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" event={"ID":"cde3f988-8444-42aa-8d40-64830292a2a3","Type":"ContainerStarted","Data":"2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1"} Nov 03 19:33:44.533951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:44.533870 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:44.555752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:44.555708 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" podStartSLOduration=1.899935862 podStartE2EDuration="4.555694289s" totalImagesPullingTime="2.655758427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:44.555428114 +0000 UTC m=+390.896541623" watchObservedRunningTime="2026-11-03 19:33:44.555694289 +0000 UTC m=+390.896807774" Nov 03 19:33:44.575660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:44.575621 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" podStartSLOduration=2.043910641 podStartE2EDuration="3.575607316s" totalImagesPullingTime="1.531696675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:44.575067319 +0000 UTC m=+390.916180835" watchObservedRunningTime="2026-11-03 19:33:44.575607316 +0000 UTC m=+390.916720846" Nov 03 19:33:51.275365 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:33:51.275326 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:33:52.522583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:52.522551 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 19:33:55.539668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:55.539599 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:57.286436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.286400 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:57.286831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.286626 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" podUID="cde3f988-8444-42aa-8d40-64830292a2a3" containerName="limitador" containerID="cri-o://2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1" gracePeriod=30 Nov 03 19:33:57.830769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.830745 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:57.900466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.900398 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr\" (UniqueName: \"kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr\") pod \"cde3f988-8444-42aa-8d40-64830292a2a3\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " Nov 03 19:33:57.900466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.900440 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file\" (UniqueName: \"kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file\") pod \"cde3f988-8444-42aa-8d40-64830292a2a3\" (UID: \"cde3f988-8444-42aa-8d40-64830292a2a3\") " Nov 03 19:33:57.900781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.900764 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file" pod "cde3f988-8444-42aa-8d40-64830292a2a3" (UID: "cde3f988-8444-42aa-8d40-64830292a2a3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:57.900872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.900850 5233 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cde3f988-8444-42aa-8d40-64830292a2a3-config-file\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:57.902661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:57.902632 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr" pod "cde3f988-8444-42aa-8d40-64830292a2a3" (UID: "cde3f988-8444-42aa-8d40-64830292a2a3"). InnerVolumeSpecName "kube-api-access-rmgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:58.002173 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.002137 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgcr\" (UniqueName: \"kubernetes.io/projected/cde3f988-8444-42aa-8d40-64830292a2a3-kube-api-access-rmgcr\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:58.604633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.604600 5233 generic.go:356] "Generic (PLEG): container finished" podID="cde3f988-8444-42aa-8d40-64830292a2a3" containerID="2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1" exitCode=0 Nov 03 19:33:58.605058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.604644 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" event={"ID":"cde3f988-8444-42aa-8d40-64830292a2a3","Type":"ContainerDied","Data":"2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1"} Nov 03 19:33:58.605058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.604683 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" event={"ID":"cde3f988-8444-42aa-8d40-64830292a2a3","Type":"ContainerDied","Data":"798779421be398a757d43c3d99df436a05adc83d9be3f0f725e69ef5a7696e56"} Nov 03 19:33:58.605058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.604703 5233 scope.go:122] "RemoveContainer" containerID="2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1" Nov 03 19:33:58.605058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.604647 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zh594" Nov 03 19:33:58.616730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.616713 5233 scope.go:122] "RemoveContainer" containerID="2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1" Nov 03 19:33:58.617023 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:33:58.617001 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1\": container with ID starting with 2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1 not found: ID does not exist" containerID="2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1" Nov 03 19:33:58.617079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.617034 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1"} err="failed to get container status \"2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1\": rpc error: code = NotFound desc = could not find container \"2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1\": container with ID starting with 2556d4815448bf0678a0d3688e966e2a0f4552728c5c7ff41c047e8f3ed26fd1 not found: ID does not exist" Nov 03 19:33:58.632025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.632002 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:58.633696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.633674 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nxx99"] Nov 03 19:33:58.634042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.634030 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cde3f988-8444-42aa-8d40-64830292a2a3" containerName="limitador" Nov 03 19:33:58.634093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.634044 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde3f988-8444-42aa-8d40-64830292a2a3" containerName="limitador" Nov 03 19:33:58.634133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.634124 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde3f988-8444-42aa-8d40-64830292a2a3" containerName="limitador" Nov 03 19:33:58.640521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.640497 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zh594"] Nov 03 19:33:58.640622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.640596 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:33:58.642085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.642064 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nxx99"] Nov 03 19:33:58.650649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.650633 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84fhw\"" Nov 03 19:33:58.707395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.707369 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq57p\" (UniqueName: \"kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p\") pod \"authorino-57f7f8659f-nxx99\" (UID: \"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5\") " pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:33:58.794423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.794396 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7xht"] Nov 03 19:33:58.798732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.798714 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:33:58.807433 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.807413 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7xht"] Nov 03 19:33:58.808138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.808113 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq57p\" (UniqueName: \"kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p\") pod \"authorino-57f7f8659f-nxx99\" (UID: \"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5\") " pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:33:58.821781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.821758 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq57p\" (UniqueName: \"kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p\") pod \"authorino-57f7f8659f-nxx99\" (UID: \"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5\") " pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:33:58.908925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.908854 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjtl\" (UniqueName: \"kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl\") pod \"authorino-56c6c9ddd5-z7xht\" (UID: \"6664055e-7b36-4b2a-98ee-d71e81d49485\") " pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:33:58.955449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:58.955425 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:33:59.010599 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.009816 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjtl\" (UniqueName: \"kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl\") pod \"authorino-56c6c9ddd5-z7xht\" (UID: \"6664055e-7b36-4b2a-98ee-d71e81d49485\") " pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:33:59.022327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.022302 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjtl\" (UniqueName: \"kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl\") pod \"authorino-56c6c9ddd5-z7xht\" (UID: \"6664055e-7b36-4b2a-98ee-d71e81d49485\") " pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:33:59.112939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.112910 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:33:59.231705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.231679 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nxx99"] Nov 03 19:33:59.234211 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:59.234181 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5789c9af_dfdd_41d4_af4c_12b3bb89e3f5.slice/crio-1650d456880811ead7fca43807f4640403982730d3cf9a52883f94f86d07ba4f WatchSource:0}: Error finding container 1650d456880811ead7fca43807f4640403982730d3cf9a52883f94f86d07ba4f: Status 404 returned error can't find the container with id 1650d456880811ead7fca43807f4640403982730d3cf9a52883f94f86d07ba4f Nov 03 19:33:59.385347 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.385317 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7xht"] Nov 03 19:33:59.387794 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:33:59.387767 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6664055e_7b36_4b2a_98ee_d71e81d49485.slice/crio-c80922e4bb5bb57ae750b2a71f89f1b035544a87f85a22df63373a08893c931d WatchSource:0}: Error finding container c80922e4bb5bb57ae750b2a71f89f1b035544a87f85a22df63373a08893c931d: Status 404 returned error can't find the container with id c80922e4bb5bb57ae750b2a71f89f1b035544a87f85a22df63373a08893c931d Nov 03 19:33:59.610741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.610710 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nxx99" event={"ID":"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5","Type":"ContainerStarted","Data":"1650d456880811ead7fca43807f4640403982730d3cf9a52883f94f86d07ba4f"} Nov 03 19:33:59.612904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.612878 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" event={"ID":"6664055e-7b36-4b2a-98ee-d71e81d49485","Type":"ContainerStarted","Data":"c80922e4bb5bb57ae750b2a71f89f1b035544a87f85a22df63373a08893c931d"} Nov 03 19:33:59.834341 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:33:59.834311 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde3f988-8444-42aa-8d40-64830292a2a3" path="/var/lib/kubelet/pods/cde3f988-8444-42aa-8d40-64830292a2a3/volumes" Nov 03 19:34:02.632822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:02.632783 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" event={"ID":"6664055e-7b36-4b2a-98ee-d71e81d49485","Type":"ContainerStarted","Data":"9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48"} Nov 03 19:34:02.634278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:02.634250 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nxx99" event={"ID":"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5","Type":"ContainerStarted","Data":"dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b"} Nov 03 19:34:02.652219 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:02.652175 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" podStartSLOduration=2.000626263 podStartE2EDuration="4.652160548s" totalImagesPullingTime="2.651534285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:02.649489581 +0000 UTC m=+408.990603070" watchObservedRunningTime="2026-11-03 19:34:02.652160548 +0000 UTC m=+408.993274033" Nov 03 19:34:02.669698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:02.669640 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nxx99" podStartSLOduration=2.358648392 podStartE2EDuration="4.669620622s" totalImagesPullingTime="2.31097223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:02.667620937 +0000 UTC m=+409.008734424" watchObservedRunningTime="2026-11-03 19:34:02.669620622 +0000 UTC m=+409.010734107" Nov 03 19:34:02.678211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:02.678182 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nxx99"] Nov 03 19:34:04.644178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:04.644141 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nxx99" podUID="5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" containerName="authorino" containerID="cri-o://dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b" gracePeriod=30 Nov 03 19:34:04.891866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:04.891840 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:34:04.960262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:04.960188 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p\" (UniqueName: \"kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p\") pod \"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5\" (UID: \"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5\") " Nov 03 19:34:04.962176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:04.962149 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p" pod "5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" (UID: "5789c9af-dfdd-41d4-af4c-12b3bb89e3f5"). InnerVolumeSpecName "kube-api-access-gq57p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:34:05.061067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.061035 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq57p\" (UniqueName: \"kubernetes.io/projected/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5-kube-api-access-gq57p\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:34:05.306965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.306930 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf"] Nov 03 19:34:05.307311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.307298 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" containerName="authorino" Nov 03 19:34:05.307360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.307311 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" containerName="authorino" Nov 03 19:34:05.307394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.307377 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" containerName="authorino" Nov 03 19:34:05.310596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.310575 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.318120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.318095 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 19:34:05.318243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.318166 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:34:05.318243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.318169 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:34:05.318243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.318236 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gs48d\"" Nov 03 19:34:05.318551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.318533 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf"] Nov 03 19:34:05.464484 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.464453 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a6e7a83-1671-4dc3-b972-a41676ac43a8-data\") pod \"postgres-7cdbf6dc44-m8bbf\" (UID: \"4a6e7a83-1671-4dc3-b972-a41676ac43a8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.464654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.464588 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8j8\" (UniqueName: \"kubernetes.io/projected/4a6e7a83-1671-4dc3-b972-a41676ac43a8-kube-api-access-7q8j8\") pod \"postgres-7cdbf6dc44-m8bbf\" (UID: \"4a6e7a83-1671-4dc3-b972-a41676ac43a8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.540104 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.540079 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mjtbj" Nov 03 19:34:05.565694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.565616 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a6e7a83-1671-4dc3-b972-a41676ac43a8-data\") pod \"postgres-7cdbf6dc44-m8bbf\" (UID: \"4a6e7a83-1671-4dc3-b972-a41676ac43a8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.565826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.565724 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j8\" (UniqueName: \"kubernetes.io/projected/4a6e7a83-1671-4dc3-b972-a41676ac43a8-kube-api-access-7q8j8\") pod \"postgres-7cdbf6dc44-m8bbf\" (UID: \"4a6e7a83-1671-4dc3-b972-a41676ac43a8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.566021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.566001 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a6e7a83-1671-4dc3-b972-a41676ac43a8-data\") pod \"postgres-7cdbf6dc44-m8bbf\" (UID: \"4a6e7a83-1671-4dc3-b972-a41676ac43a8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.579851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.579829 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8j8\" (UniqueName: \"kubernetes.io/projected/4a6e7a83-1671-4dc3-b972-a41676ac43a8-kube-api-access-7q8j8\") pod \"postgres-7cdbf6dc44-m8bbf\" (UID: \"4a6e7a83-1671-4dc3-b972-a41676ac43a8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.624723 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.624699 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:05.650661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.650635 5233 generic.go:356] "Generic (PLEG): container finished" podID="5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" containerID="dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b" exitCode=0 Nov 03 19:34:05.651047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.650691 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nxx99" Nov 03 19:34:05.651047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.650703 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nxx99" event={"ID":"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5","Type":"ContainerDied","Data":"dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b"} Nov 03 19:34:05.651047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.650746 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nxx99" event={"ID":"5789c9af-dfdd-41d4-af4c-12b3bb89e3f5","Type":"ContainerDied","Data":"1650d456880811ead7fca43807f4640403982730d3cf9a52883f94f86d07ba4f"} Nov 03 19:34:05.651047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.650764 5233 scope.go:122] "RemoveContainer" containerID="dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b" Nov 03 19:34:05.664275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.664253 5233 scope.go:122] "RemoveContainer" containerID="dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b" Nov 03 19:34:05.664573 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:34:05.664549 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b\": container with ID starting with dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b not found: ID does not exist" containerID="dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b" Nov 03 19:34:05.664657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.664586 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b"} err="failed to get container status \"dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b\": rpc error: code = NotFound desc = could not find container \"dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b\": container with ID starting with dc7d0219346671be16a92e8e32e0261ac5880c134670bf3fa41980095b30a56b not found: ID does not exist" Nov 03 19:34:05.682681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.682646 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nxx99"] Nov 03 19:34:05.694633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.691125 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nxx99"] Nov 03 19:34:05.834267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.834192 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5789c9af-dfdd-41d4-af4c-12b3bb89e3f5" path="/var/lib/kubelet/pods/5789c9af-dfdd-41d4-af4c-12b3bb89e3f5/volumes" Nov 03 19:34:05.904249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:05.904206 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf"] Nov 03 19:34:05.906285 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:34:05.906255 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6e7a83_1671_4dc3_b972_a41676ac43a8.slice/crio-fdc8c22261d16d455d2cc49848f7ce01069db841131bfcbc9779bcafd96639a9 WatchSource:0}: Error finding container fdc8c22261d16d455d2cc49848f7ce01069db841131bfcbc9779bcafd96639a9: Status 404 returned error can't find the container with id fdc8c22261d16d455d2cc49848f7ce01069db841131bfcbc9779bcafd96639a9 Nov 03 19:34:06.657703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:06.657664 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" event={"ID":"4a6e7a83-1671-4dc3-b972-a41676ac43a8","Type":"ContainerStarted","Data":"fdc8c22261d16d455d2cc49848f7ce01069db841131bfcbc9779bcafd96639a9"} Nov 03 19:34:10.973921 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.973861 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq"] Nov 03 19:34:10.981310 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.981280 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:10.985831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.985809 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq"] Nov 03 19:34:10.988024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.988004 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 19:34:10.988154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.988050 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:34:10.988154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.988098 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 19:34:10.988294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.988273 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 19:34:10.988404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.988296 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 19:34:10.988404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:10.988385 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lvbq4\"" Nov 03 19:34:11.122401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.122367 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d798784-e901-4515-9385-47efdb181643-metrics-certs\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.122401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.122414 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d798784-e901-4515-9385-47efdb181643-cert\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.122640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.122578 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d798784-e901-4515-9385-47efdb181643-manager-config\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.122640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.122620 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26986\" (UniqueName: \"kubernetes.io/projected/1d798784-e901-4515-9385-47efdb181643-kube-api-access-26986\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.223114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.223078 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d798784-e901-4515-9385-47efdb181643-manager-config\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.223316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.223141 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26986\" (UniqueName: \"kubernetes.io/projected/1d798784-e901-4515-9385-47efdb181643-kube-api-access-26986\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.223316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.223203 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d798784-e901-4515-9385-47efdb181643-metrics-certs\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.223316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.223236 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d798784-e901-4515-9385-47efdb181643-cert\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.223868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.223817 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d798784-e901-4515-9385-47efdb181643-manager-config\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.226413 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.226354 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d798784-e901-4515-9385-47efdb181643-cert\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.227632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.227608 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d798784-e901-4515-9385-47efdb181643-metrics-certs\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.239008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.238982 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26986\" (UniqueName: \"kubernetes.io/projected/1d798784-e901-4515-9385-47efdb181643-kube-api-access-26986\") pod \"jobset-controller-manager-864947c86-tbcbq\" (UID: \"1d798784-e901-4515-9385-47efdb181643\") " pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.301065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.301030 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:11.593425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.593397 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq"] Nov 03 19:34:11.594864 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:34:11.594835 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d798784_e901_4515_9385_47efdb181643.slice/crio-eca78e8fa9ae796524e7c4b982d53f7a1a280beeabb283e1c9f6901ae90f4dbc WatchSource:0}: Error finding container eca78e8fa9ae796524e7c4b982d53f7a1a280beeabb283e1c9f6901ae90f4dbc: Status 404 returned error can't find the container with id eca78e8fa9ae796524e7c4b982d53f7a1a280beeabb283e1c9f6901ae90f4dbc Nov 03 19:34:11.689041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.689006 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" event={"ID":"1d798784-e901-4515-9385-47efdb181643","Type":"ContainerStarted","Data":"eca78e8fa9ae796524e7c4b982d53f7a1a280beeabb283e1c9f6901ae90f4dbc"} Nov 03 19:34:11.690838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.690810 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" event={"ID":"4a6e7a83-1671-4dc3-b972-a41676ac43a8","Type":"ContainerStarted","Data":"1b9849e04648ea9adcb8c421b6239f6db2d2a767f921db941eaf7437bc395352"} Nov 03 19:34:11.690988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.690916 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:11.710374 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:11.710333 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" podStartSLOduration=1.128165033 podStartE2EDuration="6.71032163s" totalImagesPullingTime="5.582156597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:11.708448292 +0000 UTC m=+418.049561778" watchObservedRunningTime="2026-11-03 19:34:11.71032163 +0000 UTC m=+418.051435115" Nov 03 19:34:15.715746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:15.715703 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" event={"ID":"1d798784-e901-4515-9385-47efdb181643","Type":"ContainerStarted","Data":"003c7ca287f3a0add983f6983e080a76d43721b4d359884f663cfc893f5f6e11"} Nov 03 19:34:15.716146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:15.716090 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:15.736854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:15.736809 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" podStartSLOduration=1.950695904 podStartE2EDuration="5.736795134s" totalImagesPullingTime="3.78609923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:15.735210236 +0000 UTC m=+422.076323721" watchObservedRunningTime="2026-11-03 19:34:15.736795134 +0000 UTC m=+422.077908624" Nov 03 19:34:17.726404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:17.726371 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m8bbf" Nov 03 19:34:21.276285 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:34:21.276245 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:34:27.731646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:34:27.731619 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-864947c86-tbcbq" Nov 03 19:34:51.274987 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:34:51.274933 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:35:12.485670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.485633 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5qd7g"] Nov 03 19:35:12.488999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.488934 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:12.497925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.497897 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5qd7g"] Nov 03 19:35:12.589000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.588956 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5qd7g"] Nov 03 19:35:12.589196 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:35:12.589177 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7lrs4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" podUID="552f5f1b-940e-41c2-ad8a-9f1b760583c0" Nov 03 19:35:12.591244 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.591217 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrs4\" (UniqueName: \"kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4\") pod \"authorino-6f6944bcdd-5qd7g\" (UID: \"552f5f1b-940e-41c2-ad8a-9f1b760583c0\") " pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:12.625511 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.625484 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6jh6"] Nov 03 19:35:12.629095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.629077 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.640470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.640452 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 19:35:12.641679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.641660 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6jh6"] Nov 03 19:35:12.692033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.692004 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrs4\" (UniqueName: \"kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4\") pod \"authorino-6f6944bcdd-5qd7g\" (UID: \"552f5f1b-940e-41c2-ad8a-9f1b760583c0\") " pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:12.692176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.692072 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert\") pod \"authorino-6889cbd8c4-p6jh6\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.692176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.692098 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49ln\" (UniqueName: \"kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln\") pod \"authorino-6889cbd8c4-p6jh6\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.715606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.715574 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrs4\" (UniqueName: \"kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4\") pod \"authorino-6f6944bcdd-5qd7g\" (UID: \"552f5f1b-940e-41c2-ad8a-9f1b760583c0\") " pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:12.793479 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.793397 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert\") pod \"authorino-6889cbd8c4-p6jh6\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.793479 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.793434 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49ln\" (UniqueName: \"kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln\") pod \"authorino-6889cbd8c4-p6jh6\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.795719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.795699 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert\") pod \"authorino-6889cbd8c4-p6jh6\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.810763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.810738 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49ln\" (UniqueName: \"kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln\") pod \"authorino-6889cbd8c4-p6jh6\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.881748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.881718 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6jh6"] Nov 03 19:35:12.881934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.881923 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:12.928505 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.928472 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58d476ffc7-qkvn9"] Nov 03 19:35:12.932723 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.932700 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:12.951089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.951025 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58d476ffc7-qkvn9"] Nov 03 19:35:12.994600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.994568 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert\") pod \"authorino-58d476ffc7-qkvn9\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:12.994600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.994580 5233 generic.go:356] "Generic (PLEG): container finished" podID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerID="877215498c1d73080faae0c4f4a2fbcf5f15415a30a00f09bee1aa1525047617" exitCode=0 Nov 03 19:35:12.994794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.994648 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" event={"ID":"d6929ca2-35e3-40c0-8010-49b9dc0a68c5","Type":"ContainerDied","Data":"877215498c1d73080faae0c4f4a2fbcf5f15415a30a00f09bee1aa1525047617"} Nov 03 19:35:12.994794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.994673 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:12.994794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.994703 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d472s\" (UniqueName: \"kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s\") pod \"authorino-58d476ffc7-qkvn9\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:12.994997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.994965 5233 scope.go:122] "RemoveContainer" containerID="877215498c1d73080faae0c4f4a2fbcf5f15415a30a00f09bee1aa1525047617" Nov 03 19:35:12.996116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.996085 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:12.996202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.996163 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:12.996202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:12.996197 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:35:13.005777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.005759 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:13.096262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.095306 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4\" (UniqueName: \"kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4\") pod \"552f5f1b-940e-41c2-ad8a-9f1b760583c0\" (UID: \"552f5f1b-940e-41c2-ad8a-9f1b760583c0\") " Nov 03 19:35:13.096262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.095586 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d472s\" (UniqueName: \"kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s\") pod \"authorino-58d476ffc7-qkvn9\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:13.096262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.095677 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert\") pod \"authorino-58d476ffc7-qkvn9\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:13.098564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.098539 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert\") pod \"authorino-58d476ffc7-qkvn9\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:13.098708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.098615 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4" pod "552f5f1b-940e-41c2-ad8a-9f1b760583c0" (UID: "552f5f1b-940e-41c2-ad8a-9f1b760583c0"). InnerVolumeSpecName "kube-api-access-7lrs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:13.109408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.109379 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d472s\" (UniqueName: \"kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s\") pod \"authorino-58d476ffc7-qkvn9\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:13.168516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.168488 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6jh6"] Nov 03 19:35:13.170758 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:35:13.170728 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4352bf67_c562_4060_98a9_d4851b4102be.slice/crio-5a1df1eaf4bf037e3ec736d9526f25ba8dbe2b2178dc416e621a8bec38285d0a WatchSource:0}: Error finding container 5a1df1eaf4bf037e3ec736d9526f25ba8dbe2b2178dc416e621a8bec38285d0a: Status 404 returned error can't find the container with id 5a1df1eaf4bf037e3ec736d9526f25ba8dbe2b2178dc416e621a8bec38285d0a Nov 03 19:35:13.196813 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.196788 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrs4\" (UniqueName: \"kubernetes.io/projected/552f5f1b-940e-41c2-ad8a-9f1b760583c0-kube-api-access-7lrs4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:13.255849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.255820 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:35:13.544123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:13.544098 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58d476ffc7-qkvn9"] Nov 03 19:35:13.546552 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:35:13.546530 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc2a2853_4228_425b_9ecb_e6b57b3f1f61.slice/crio-30c2ad95d43a7a5390ca6e0565edef7badb87e3ef503b1b6d91f1a6de80df1b6 WatchSource:0}: Error finding container 30c2ad95d43a7a5390ca6e0565edef7badb87e3ef503b1b6d91f1a6de80df1b6: Status 404 returned error can't find the container with id 30c2ad95d43a7a5390ca6e0565edef7badb87e3ef503b1b6d91f1a6de80df1b6 Nov 03 19:35:14.015670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.015632 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" event={"ID":"d6929ca2-35e3-40c0-8010-49b9dc0a68c5","Type":"ContainerStarted","Data":"d568157f6f38d77ba967b2dde16e6504fa7a6ee76bcef906ca576f3202895f02"} Nov 03 19:35:14.016829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.016800 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" event={"ID":"fc2a2853-4228-425b-9ecb-e6b57b3f1f61","Type":"ContainerStarted","Data":"30c2ad95d43a7a5390ca6e0565edef7badb87e3ef503b1b6d91f1a6de80df1b6"} Nov 03 19:35:14.017836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.017813 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" event={"ID":"4352bf67-c562-4060-98a9-d4851b4102be","Type":"ContainerStarted","Data":"5a1df1eaf4bf037e3ec736d9526f25ba8dbe2b2178dc416e621a8bec38285d0a"} Nov 03 19:35:14.017836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.017830 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5qd7g" Nov 03 19:35:14.122182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.122080 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5qd7g"] Nov 03 19:35:14.123755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.123730 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5qd7g"] Nov 03 19:35:14.323017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.322993 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:35:14.323573 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.323545 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Nov 03 19:35:14.323683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:14.323599 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Nov 03 19:35:15.028192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.028139 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" event={"ID":"fc2a2853-4228-425b-9ecb-e6b57b3f1f61","Type":"ContainerStarted","Data":"340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613"} Nov 03 19:35:15.031238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.031051 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" event={"ID":"4352bf67-c562-4060-98a9-d4851b4102be","Type":"ContainerStarted","Data":"420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea"} Nov 03 19:35:15.031238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.031102 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" podUID="4352bf67-c562-4060-98a9-d4851b4102be" containerName="authorino" containerID="cri-o://420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea" gracePeriod=30 Nov 03 19:35:15.057377 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.057312 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" podStartSLOduration=2.333915238 podStartE2EDuration="3.057292037s" totalImagesPullingTime="723.376799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:15.055987439 +0000 UTC m=+481.397100923" watchObservedRunningTime="2026-11-03 19:35:15.057292037 +0000 UTC m=+481.398405523" Nov 03 19:35:15.084388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.084265 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" podStartSLOduration=1.815072889 podStartE2EDuration="3.084244304s" totalImagesPullingTime="1.269171415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:15.081858632 +0000 UTC m=+481.422972120" watchObservedRunningTime="2026-11-03 19:35:15.084244304 +0000 UTC m=+481.425357790" Nov 03 19:35:15.119331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.119296 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7xht"] Nov 03 19:35:15.119758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.119723 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" podUID="6664055e-7b36-4b2a-98ee-d71e81d49485" containerName="authorino" containerID="cri-o://9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48" gracePeriod=30 Nov 03 19:35:15.328992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.326294 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Nov 03 19:35:15.328992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.326352 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Nov 03 19:35:15.424139 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.424108 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:15.451899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.451843 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:35:15.535520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.535487 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln\" (UniqueName: \"kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln\") pod \"4352bf67-c562-4060-98a9-d4851b4102be\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " Nov 03 19:35:15.535702 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.535677 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl\" (UniqueName: \"kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl\") pod \"6664055e-7b36-4b2a-98ee-d71e81d49485\" (UID: \"6664055e-7b36-4b2a-98ee-d71e81d49485\") " Nov 03 19:35:15.535767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.535754 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert\" (UniqueName: \"kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert\") pod \"4352bf67-c562-4060-98a9-d4851b4102be\" (UID: \"4352bf67-c562-4060-98a9-d4851b4102be\") " Nov 03 19:35:15.537539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.537515 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln" pod "4352bf67-c562-4060-98a9-d4851b4102be" (UID: "4352bf67-c562-4060-98a9-d4851b4102be"). InnerVolumeSpecName "kube-api-access-r49ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:15.537646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.537621 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl" pod "6664055e-7b36-4b2a-98ee-d71e81d49485" (UID: "6664055e-7b36-4b2a-98ee-d71e81d49485"). InnerVolumeSpecName "kube-api-access-dxjtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:35:15.545917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.545894 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert" pod "4352bf67-c562-4060-98a9-d4851b4102be" (UID: "4352bf67-c562-4060-98a9-d4851b4102be"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:35:15.637353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.637274 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49ln\" (UniqueName: \"kubernetes.io/projected/4352bf67-c562-4060-98a9-d4851b4102be-kube-api-access-r49ln\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:15.637353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.637302 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjtl\" (UniqueName: \"kubernetes.io/projected/6664055e-7b36-4b2a-98ee-d71e81d49485-kube-api-access-dxjtl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:15.637353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.637312 5233 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4352bf67-c562-4060-98a9-d4851b4102be-tls-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:35:15.834634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:15.834596 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552f5f1b-940e-41c2-ad8a-9f1b760583c0" path="/var/lib/kubelet/pods/552f5f1b-940e-41c2-ad8a-9f1b760583c0/volumes" Nov 03 19:35:16.036759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.036721 5233 generic.go:356] "Generic (PLEG): container finished" podID="6664055e-7b36-4b2a-98ee-d71e81d49485" containerID="9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48" exitCode=0 Nov 03 19:35:16.037208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.036768 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" Nov 03 19:35:16.037208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.036803 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" event={"ID":"6664055e-7b36-4b2a-98ee-d71e81d49485","Type":"ContainerDied","Data":"9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48"} Nov 03 19:35:16.037208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.036848 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z7xht" event={"ID":"6664055e-7b36-4b2a-98ee-d71e81d49485","Type":"ContainerDied","Data":"c80922e4bb5bb57ae750b2a71f89f1b035544a87f85a22df63373a08893c931d"} Nov 03 19:35:16.037208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.036860 5233 scope.go:122] "RemoveContainer" containerID="9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48" Nov 03 19:35:16.038283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.038263 5233 generic.go:356] "Generic (PLEG): container finished" podID="4352bf67-c562-4060-98a9-d4851b4102be" containerID="420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea" exitCode=0 Nov 03 19:35:16.038382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.038305 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" Nov 03 19:35:16.038382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.038352 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" event={"ID":"4352bf67-c562-4060-98a9-d4851b4102be","Type":"ContainerDied","Data":"420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea"} Nov 03 19:35:16.038460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.038400 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p6jh6" event={"ID":"4352bf67-c562-4060-98a9-d4851b4102be","Type":"ContainerDied","Data":"5a1df1eaf4bf037e3ec736d9526f25ba8dbe2b2178dc416e621a8bec38285d0a"} Nov 03 19:35:16.054220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.054199 5233 scope.go:122] "RemoveContainer" containerID="9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48" Nov 03 19:35:16.054476 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:35:16.054460 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48\": container with ID starting with 9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48 not found: ID does not exist" containerID="9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48" Nov 03 19:35:16.054542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.054483 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48"} err="failed to get container status \"9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48\": rpc error: code = NotFound desc = could not find container \"9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48\": container with ID starting with 9cc2f1aebe1bf762fce9bcbd592aa311ef948bc2f7dd8c691a1b92cb9addba48 not found: ID does not exist" Nov 03 19:35:16.054542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.054498 5233 scope.go:122] "RemoveContainer" containerID="420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea" Nov 03 19:35:16.062051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.062035 5233 scope.go:122] "RemoveContainer" containerID="420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea" Nov 03 19:35:16.062329 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:35:16.062312 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea\": container with ID starting with 420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea not found: ID does not exist" containerID="420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea" Nov 03 19:35:16.062377 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.062335 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea"} err="failed to get container status \"420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea\": rpc error: code = NotFound desc = could not find container \"420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea\": container with ID starting with 420f1cddb7d80ebad2eba96a7a9956598004ca7aee79028526875331af89c0ea not found: ID does not exist" Nov 03 19:35:16.067197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.067178 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7xht"] Nov 03 19:35:16.071703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.071684 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z7xht"] Nov 03 19:35:16.112789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.112762 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6jh6"] Nov 03 19:35:16.114335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.114316 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6jh6"] Nov 03 19:35:16.322641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.322566 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Nov 03 19:35:16.322641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:16.322618 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Nov 03 19:35:17.323176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:17.323139 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Nov 03 19:35:17.323649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:17.323195 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Nov 03 19:35:17.835321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:17.835282 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4352bf67-c562-4060-98a9-d4851b4102be" path="/var/lib/kubelet/pods/4352bf67-c562-4060-98a9-d4851b4102be/volumes" Nov 03 19:35:17.835757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:17.835738 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6664055e-7b36-4b2a-98ee-d71e81d49485" path="/var/lib/kubelet/pods/6664055e-7b36-4b2a-98ee-d71e81d49485/volumes" Nov 03 19:35:18.323037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:18.323000 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Nov 03 19:35:18.323215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:18.323053 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Nov 03 19:35:19.117625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:19.117584 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:35:19.326415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:19.326382 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:35:20.060097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:35:20.060070 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" Nov 03 19:35:21.275656 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:35:21.275621 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:35:51.275494 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:35:51.275409 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:36:21.275850 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:36:21.275808 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:36:29.670822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.670779 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk"] Nov 03 19:36:29.671448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.671189 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6664055e-7b36-4b2a-98ee-d71e81d49485" containerName="authorino" Nov 03 19:36:29.671448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.671202 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6664055e-7b36-4b2a-98ee-d71e81d49485" containerName="authorino" Nov 03 19:36:29.671448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.671232 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4352bf67-c562-4060-98a9-d4851b4102be" containerName="authorino" Nov 03 19:36:29.671448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.671238 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4352bf67-c562-4060-98a9-d4851b4102be" containerName="authorino" Nov 03 19:36:29.671448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.671290 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6664055e-7b36-4b2a-98ee-d71e81d49485" containerName="authorino" Nov 03 19:36:29.671448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.671302 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4352bf67-c562-4060-98a9-d4851b4102be" containerName="authorino" Nov 03 19:36:29.674935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.674909 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.685928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.685903 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 19:36:29.686293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.686273 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bfhff\"" Nov 03 19:36:29.686399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.686322 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 19:36:29.686399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.686348 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 19:36:29.686755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.686735 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk"] Nov 03 19:36:29.779998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.779943 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.779998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.779997 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.780201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.780061 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.780201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.780183 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7mk\" (UniqueName: \"kubernetes.io/projected/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-kube-api-access-2k7mk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.780275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.780217 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.780275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.780241 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881384 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881346 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881392 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881449 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881478 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881731 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881578 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881731 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881691 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7mk\" (UniqueName: \"kubernetes.io/projected/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-kube-api-access-2k7mk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881856 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.881946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.881905 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.882120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.882101 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.883661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.883634 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.883956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.883936 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.893962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.893941 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7mk\" (UniqueName: \"kubernetes.io/projected/cdb9e282-b5b5-457f-bbae-56f64d69ff9a-kube-api-access-2k7mk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-852wk\" (UID: \"cdb9e282-b5b5-457f-bbae-56f64d69ff9a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:29.989158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:29.989127 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:30.268790 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:30.268764 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk"] Nov 03 19:36:30.271296 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:36:30.271267 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb9e282_b5b5_457f_bbae_56f64d69ff9a.slice/crio-148484082ee0767829858c39a323f40d912a39b575a634d048bbe550a91e57aa WatchSource:0}: Error finding container 148484082ee0767829858c39a323f40d912a39b575a634d048bbe550a91e57aa: Status 404 returned error can't find the container with id 148484082ee0767829858c39a323f40d912a39b575a634d048bbe550a91e57aa Nov 03 19:36:30.403043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:30.402995 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" event={"ID":"cdb9e282-b5b5-457f-bbae-56f64d69ff9a","Type":"ContainerStarted","Data":"148484082ee0767829858c39a323f40d912a39b575a634d048bbe550a91e57aa"} Nov 03 19:36:32.419799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.419751 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" event={"ID":"cdb9e282-b5b5-457f-bbae-56f64d69ff9a","Type":"ContainerStarted","Data":"576d1f9654d94840f28bb7a38863a4459b8392d2105e5661e9a756c498dc4189"} Nov 03 19:36:32.443562 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.443471 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" podStartSLOduration=1.854338686 podStartE2EDuration="3.443452065s" totalImagesPullingTime="1.589113379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:32.442419173 +0000 UTC m=+558.783532659" watchObservedRunningTime="2026-11-03 19:36:32.443452065 +0000 UTC m=+558.784565553" Nov 03 19:36:32.521524 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.521491 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc"] Nov 03 19:36:32.528330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.528303 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.530900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.530774 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc"] Nov 03 19:36:32.538775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.538743 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 19:36:32.618227 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.618183 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.618419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.618350 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.618419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.618389 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.618539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.618474 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.618539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.618514 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvg84\" (UniqueName: \"kubernetes.io/projected/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-kube-api-access-bvg84\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.618649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.618580 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.719771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.719727 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.719950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.719776 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.719950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.719807 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.719950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.719832 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg84\" (UniqueName: \"kubernetes.io/projected/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-kube-api-access-bvg84\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.719950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.719861 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.720241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.719963 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.720241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.720182 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.720350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.720238 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.720775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.720748 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.722489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.722446 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.723093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.723071 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.733846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.733795 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg84\" (UniqueName: \"kubernetes.io/projected/92edadb3-9fae-48a3-aed1-54b25cf7c0ab-kube-api-access-bvg84\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qvdpc\" (UID: \"92edadb3-9fae-48a3-aed1-54b25cf7c0ab\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.860155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.860116 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:32.989447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:32.989370 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:33.140943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:33.140917 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc"] Nov 03 19:36:33.143036 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:36:33.143001 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92edadb3_9fae_48a3_aed1_54b25cf7c0ab.slice/crio-96a07dba28cde7388eba21b3bbda216d95e50db769ee9c0f2f54f4131132248f WatchSource:0}: Error finding container 96a07dba28cde7388eba21b3bbda216d95e50db769ee9c0f2f54f4131132248f: Status 404 returned error can't find the container with id 96a07dba28cde7388eba21b3bbda216d95e50db769ee9c0f2f54f4131132248f Nov 03 19:36:33.428059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:33.427466 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" event={"ID":"92edadb3-9fae-48a3-aed1-54b25cf7c0ab","Type":"ContainerStarted","Data":"96a07dba28cde7388eba21b3bbda216d95e50db769ee9c0f2f54f4131132248f"} Nov 03 19:36:34.436270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:34.436221 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" event={"ID":"92edadb3-9fae-48a3-aed1-54b25cf7c0ab","Type":"ContainerStarted","Data":"d1b223b6fcee182996ed5bcd92659c42d95deae8ec9ad54538f80530c77cf58b"} Nov 03 19:36:34.461393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:34.461347 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" podStartSLOduration=2.119333693 podStartE2EDuration="2.461334855s" totalImagesPullingTime="342.001162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:34.460931799 +0000 UTC m=+560.802045283" watchObservedRunningTime="2026-11-03 19:36:34.461334855 +0000 UTC m=+560.802448340" Nov 03 19:36:34.860602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:34.860571 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:36.905587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:36.905545 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mqbt7"] Nov 03 19:36:36.925040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:36.924491 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mqbt7"] Nov 03 19:36:36.925040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:36.924630 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:36.937262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:36.936560 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 19:36:37.064765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.064724 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.064931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.064814 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-home\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.064931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.064845 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.064931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.064881 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-kube-api-access-svtcm\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.065050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.064964 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.065085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.065058 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166242 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166286 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166324 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-home\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166341 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166364 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-kube-api-access-svtcm\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166386 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166781 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-home\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.166868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.166843 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.167271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.167244 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.168704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.168685 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.169031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.169015 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.182340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.182314 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/f8189b43-c1cd-45ca-9f06-d2dd639e0fde-kube-api-access-svtcm\") pod \"echo-simulator-kserve-64b6f8f96c-mqbt7\" (UID: \"f8189b43-c1cd-45ca-9f06-d2dd639e0fde\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.244552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.244519 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:37.535019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:37.534876 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mqbt7"] Nov 03 19:36:37.537866 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:36:37.537835 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8189b43_c1cd_45ca_9f06_d2dd639e0fde.slice/crio-7704a3aecdcc932d24e50d33f525699c1be5ca0395cea93810b0af1573e506a8 WatchSource:0}: Error finding container 7704a3aecdcc932d24e50d33f525699c1be5ca0395cea93810b0af1573e506a8: Status 404 returned error can't find the container with id 7704a3aecdcc932d24e50d33f525699c1be5ca0395cea93810b0af1573e506a8 Nov 03 19:36:38.465276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:38.465241 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" event={"ID":"f8189b43-c1cd-45ca-9f06-d2dd639e0fde","Type":"ContainerStarted","Data":"7704a3aecdcc932d24e50d33f525699c1be5ca0395cea93810b0af1573e506a8"} Nov 03 19:36:39.099008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.098961 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4"] Nov 03 19:36:39.103470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.103078 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.115481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.115438 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 19:36:39.117535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.117101 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4"] Nov 03 19:36:39.187710 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.187679 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e4a136-8e90-43e8-86c5-0b666a31cd70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.187885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.187738 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.187885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.187758 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.188034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.187908 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.188034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.188006 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.188139 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.188043 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zczlq\" (UniqueName: \"kubernetes.io/projected/91e4a136-8e90-43e8-86c5-0b666a31cd70-kube-api-access-zczlq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289025 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289109 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289178 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zczlq\" (UniqueName: \"kubernetes.io/projected/91e4a136-8e90-43e8-86c5-0b666a31cd70-kube-api-access-zczlq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289206 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e4a136-8e90-43e8-86c5-0b666a31cd70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289500 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289503 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289565 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.289705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289577 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.290034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.289894 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.293048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.293027 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e4a136-8e90-43e8-86c5-0b666a31cd70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.297653 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.297632 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e4a136-8e90-43e8-86c5-0b666a31cd70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.302240 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.302197 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zczlq\" (UniqueName: \"kubernetes.io/projected/91e4a136-8e90-43e8-86c5-0b666a31cd70-kube-api-access-zczlq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4\" (UID: \"91e4a136-8e90-43e8-86c5-0b666a31cd70\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.421126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.421031 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:39.733992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.733938 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4"] Nov 03 19:36:39.737037 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:36:39.737004 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e4a136_8e90_43e8_86c5_0b666a31cd70.slice/crio-2f8d225a44c56f4952f9d8c009d8b5aa4ca325464a73018051477d230af3bb4f WatchSource:0}: Error finding container 2f8d225a44c56f4952f9d8c009d8b5aa4ca325464a73018051477d230af3bb4f: Status 404 returned error can't find the container with id 2f8d225a44c56f4952f9d8c009d8b5aa4ca325464a73018051477d230af3bb4f Nov 03 19:36:39.990644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:39.989753 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:40.014519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.014276 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:40.477827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.477789 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" event={"ID":"91e4a136-8e90-43e8-86c5-0b666a31cd70","Type":"ContainerStarted","Data":"f553bc3ee4d236cd87a96b6bdc559c1f5530ceb6d5cad19242be26812f6a8ad0"} Nov 03 19:36:40.478022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.477839 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" event={"ID":"91e4a136-8e90-43e8-86c5-0b666a31cd70","Type":"ContainerStarted","Data":"2f8d225a44c56f4952f9d8c009d8b5aa4ca325464a73018051477d230af3bb4f"} Nov 03 19:36:40.481114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.480489 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" event={"ID":"f8189b43-c1cd-45ca-9f06-d2dd639e0fde","Type":"ContainerStarted","Data":"6e61cc4370ccf197b4fa3c1fc4253d60523fa8064d2901601727ea5fb9823a85"} Nov 03 19:36:40.496480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.496414 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-852wk" Nov 03 19:36:40.527037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.526957 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podStartSLOduration=1.206252886 podStartE2EDuration="1.526937753s" totalImagesPullingTime="320.684867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:40.503314981 +0000 UTC m=+566.844428466" watchObservedRunningTime="2026-11-03 19:36:40.526937753 +0000 UTC m=+566.868051239" Nov 03 19:36:40.550062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:40.550002 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" podStartSLOduration=1.931673857 podStartE2EDuration="4.549960887s" totalImagesPullingTime="2.61828703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:40.549430309 +0000 UTC m=+566.890543795" watchObservedRunningTime="2026-11-03 19:36:40.549960887 +0000 UTC m=+566.891074375" Nov 03 19:36:41.245092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:41.245055 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:41.421449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:41.421411 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:42.860408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:42.860370 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:42.873726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:42.873699 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:43.520400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:43.520358 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" Nov 03 19:36:47.245640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:47.245607 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:47.258781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:47.258750 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:47.536619 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:47.536549 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mqbt7" Nov 03 19:36:49.422032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:49.421996 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:49.434719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:49.434695 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:49.546458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:49.546432 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:36:51.216442 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.216397 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f46bb44fd-jzdlk"] Nov 03 19:36:51.220275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.220256 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.228495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.228473 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f46bb44fd-jzdlk"] Nov 03 19:36:51.275107 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:36:51.275072 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:36:51.338160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.338124 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8szf\" (UniqueName: \"kubernetes.io/projected/c2793993-763c-4b31-b090-69ac6ea8c04e-kube-api-access-r8szf\") pod \"authorino-5f46bb44fd-jzdlk\" (UID: \"c2793993-763c-4b31-b090-69ac6ea8c04e\") " pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.338385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.338192 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2793993-763c-4b31-b090-69ac6ea8c04e-tls-cert\") pod \"authorino-5f46bb44fd-jzdlk\" (UID: \"c2793993-763c-4b31-b090-69ac6ea8c04e\") " pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.438816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.438771 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2793993-763c-4b31-b090-69ac6ea8c04e-tls-cert\") pod \"authorino-5f46bb44fd-jzdlk\" (UID: \"c2793993-763c-4b31-b090-69ac6ea8c04e\") " pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.439051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.438863 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8szf\" (UniqueName: \"kubernetes.io/projected/c2793993-763c-4b31-b090-69ac6ea8c04e-kube-api-access-r8szf\") pod \"authorino-5f46bb44fd-jzdlk\" (UID: \"c2793993-763c-4b31-b090-69ac6ea8c04e\") " pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.441257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.441230 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2793993-763c-4b31-b090-69ac6ea8c04e-tls-cert\") pod \"authorino-5f46bb44fd-jzdlk\" (UID: \"c2793993-763c-4b31-b090-69ac6ea8c04e\") " pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.452167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.452121 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8szf\" (UniqueName: \"kubernetes.io/projected/c2793993-763c-4b31-b090-69ac6ea8c04e-kube-api-access-r8szf\") pod \"authorino-5f46bb44fd-jzdlk\" (UID: \"c2793993-763c-4b31-b090-69ac6ea8c04e\") " pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.534017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.533930 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" Nov 03 19:36:51.814275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:51.814249 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f46bb44fd-jzdlk"] Nov 03 19:36:51.816716 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:36:51.816677 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2793993_763c_4b31_b090_69ac6ea8c04e.slice/crio-cdcfe059b236928c7924517c048065a95c303319329bd518e3b828a68ecec272 WatchSource:0}: Error finding container cdcfe059b236928c7924517c048065a95c303319329bd518e3b828a68ecec272: Status 404 returned error can't find the container with id cdcfe059b236928c7924517c048065a95c303319329bd518e3b828a68ecec272 Nov 03 19:36:52.553063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:52.553022 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" event={"ID":"c2793993-763c-4b31-b090-69ac6ea8c04e","Type":"ContainerStarted","Data":"cdcfe059b236928c7924517c048065a95c303319329bd518e3b828a68ecec272"} Nov 03 19:36:53.564055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.563958 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" event={"ID":"c2793993-763c-4b31-b090-69ac6ea8c04e","Type":"ContainerStarted","Data":"611e9eb8c54b869a05c12118aeb69dbc57059a189c68fc24fd3ee66b00999f9a"} Nov 03 19:36:53.585196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.585144 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f46bb44fd-jzdlk" podStartSLOduration=1.763091335 podStartE2EDuration="2.585128723s" totalImagesPullingTime="822.037388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:53.583224071 +0000 UTC m=+579.924337556" watchObservedRunningTime="2026-11-03 19:36:53.585128723 +0000 UTC m=+579.926242207" Nov 03 19:36:53.618727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.618692 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58d476ffc7-qkvn9"] Nov 03 19:36:53.618966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.618932 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" podUID="fc2a2853-4228-425b-9ecb-e6b57b3f1f61" containerName="authorino" containerID="cri-o://340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613" gracePeriod=30 Nov 03 19:36:53.888031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.888007 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:36:53.961780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.961733 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert\" (UniqueName: \"kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert\") pod \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " Nov 03 19:36:53.961961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.961803 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s\" (UniqueName: \"kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s\") pod \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\" (UID: \"fc2a2853-4228-425b-9ecb-e6b57b3f1f61\") " Nov 03 19:36:53.963985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.963936 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s" pod "fc2a2853-4228-425b-9ecb-e6b57b3f1f61" (UID: "fc2a2853-4228-425b-9ecb-e6b57b3f1f61"). InnerVolumeSpecName "kube-api-access-d472s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:53.974523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:53.974486 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert" pod "fc2a2853-4228-425b-9ecb-e6b57b3f1f61" (UID: "fc2a2853-4228-425b-9ecb-e6b57b3f1f61"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:54.074684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.074637 5233 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-tls-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:54.074868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.074687 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d472s\" (UniqueName: \"kubernetes.io/projected/fc2a2853-4228-425b-9ecb-e6b57b3f1f61-kube-api-access-d472s\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:54.571019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.570956 5233 generic.go:356] "Generic (PLEG): container finished" podID="fc2a2853-4228-425b-9ecb-e6b57b3f1f61" containerID="340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613" exitCode=0 Nov 03 19:36:54.571019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.571022 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" Nov 03 19:36:54.571536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.571051 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" event={"ID":"fc2a2853-4228-425b-9ecb-e6b57b3f1f61","Type":"ContainerDied","Data":"340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613"} Nov 03 19:36:54.571536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.571097 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d476ffc7-qkvn9" event={"ID":"fc2a2853-4228-425b-9ecb-e6b57b3f1f61","Type":"ContainerDied","Data":"30c2ad95d43a7a5390ca6e0565edef7badb87e3ef503b1b6d91f1a6de80df1b6"} Nov 03 19:36:54.571536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.571119 5233 scope.go:122] "RemoveContainer" containerID="340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613" Nov 03 19:36:54.581759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.581740 5233 scope.go:122] "RemoveContainer" containerID="340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613" Nov 03 19:36:54.582085 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:36:54.582063 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613\": container with ID starting with 340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613 not found: ID does not exist" containerID="340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613" Nov 03 19:36:54.582182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.582097 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613"} err="failed to get container status \"340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613\": rpc error: code = NotFound desc = could not find container \"340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613\": container with ID starting with 340cf3a1ff0f0c68bb08f4df0a1284a313689daed90cebba780407e76b56c613 not found: ID does not exist" Nov 03 19:36:54.596133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.596108 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58d476ffc7-qkvn9"] Nov 03 19:36:54.601521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:54.601495 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58d476ffc7-qkvn9"] Nov 03 19:36:55.837279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:36:55.837244 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2a2853-4228-425b-9ecb-e6b57b3f1f61" path="/var/lib/kubelet/pods/fc2a2853-4228-425b-9ecb-e6b57b3f1f61/volumes" Nov 03 19:37:00.527716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.527683 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2n225"] Nov 03 19:37:00.528303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.528287 5233 cpu_manager.go:414] "containerMap: removing container" podUID="fc2a2853-4228-425b-9ecb-e6b57b3f1f61" containerName="authorino" Nov 03 19:37:00.528363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.528306 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2a2853-4228-425b-9ecb-e6b57b3f1f61" containerName="authorino" Nov 03 19:37:00.528424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.528413 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2a2853-4228-425b-9ecb-e6b57b3f1f61" containerName="authorino" Nov 03 19:37:00.532026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.532003 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.542594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.542566 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2n225"] Nov 03 19:37:00.545057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.545036 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 19:37:00.545654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.545605 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rmr9c\"" Nov 03 19:37:00.562680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.562651 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-57shs"] Nov 03 19:37:00.566991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.566957 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.574300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.574270 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-57shs"] Nov 03 19:37:00.653103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.653066 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea15765-cb00-498d-a197-4550768d770b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-57shs\" (UID: \"bea15765-cb00-498d-a197-4550768d770b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.653103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.653106 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784da5d0-b4e7-4590-a365-2be891af94c2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2n225\" (UID: \"784da5d0-b4e7-4590-a365-2be891af94c2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.653468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.653128 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddp7\" (UniqueName: \"kubernetes.io/projected/bea15765-cb00-498d-a197-4550768d770b-kube-api-access-dddp7\") pod \"payload-processing-55fbbbf9dc-57shs\" (UID: \"bea15765-cb00-498d-a197-4550768d770b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.653468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.653320 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4gq\" (UniqueName: \"kubernetes.io/projected/784da5d0-b4e7-4590-a365-2be891af94c2-kube-api-access-rb4gq\") pod \"payload-pre-processing-7b98bb678c-2n225\" (UID: \"784da5d0-b4e7-4590-a365-2be891af94c2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.754403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.754365 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea15765-cb00-498d-a197-4550768d770b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-57shs\" (UID: \"bea15765-cb00-498d-a197-4550768d770b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.754403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.754403 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784da5d0-b4e7-4590-a365-2be891af94c2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2n225\" (UID: \"784da5d0-b4e7-4590-a365-2be891af94c2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.754807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.754426 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddp7\" (UniqueName: \"kubernetes.io/projected/bea15765-cb00-498d-a197-4550768d770b-kube-api-access-dddp7\") pod \"payload-processing-55fbbbf9dc-57shs\" (UID: \"bea15765-cb00-498d-a197-4550768d770b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.754807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.754468 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4gq\" (UniqueName: \"kubernetes.io/projected/784da5d0-b4e7-4590-a365-2be891af94c2-kube-api-access-rb4gq\") pod \"payload-pre-processing-7b98bb678c-2n225\" (UID: \"784da5d0-b4e7-4590-a365-2be891af94c2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.755277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.755249 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea15765-cb00-498d-a197-4550768d770b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-57shs\" (UID: \"bea15765-cb00-498d-a197-4550768d770b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.755372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.755257 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784da5d0-b4e7-4590-a365-2be891af94c2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2n225\" (UID: \"784da5d0-b4e7-4590-a365-2be891af94c2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.766985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.766944 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddp7\" (UniqueName: \"kubernetes.io/projected/bea15765-cb00-498d-a197-4550768d770b-kube-api-access-dddp7\") pod \"payload-processing-55fbbbf9dc-57shs\" (UID: \"bea15765-cb00-498d-a197-4550768d770b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:00.767094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.767021 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4gq\" (UniqueName: \"kubernetes.io/projected/784da5d0-b4e7-4590-a365-2be891af94c2-kube-api-access-rb4gq\") pod \"payload-pre-processing-7b98bb678c-2n225\" (UID: \"784da5d0-b4e7-4590-a365-2be891af94c2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.857159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.857077 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:00.882386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:00.882357 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:01.135788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:01.135764 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2n225"] Nov 03 19:37:01.137834 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:37:01.137797 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784da5d0_b4e7_4590_a365_2be891af94c2.slice/crio-5167d01538105dc526d6073ab696a5afaa6c9a4fda925b4d02b71df007726543 WatchSource:0}: Error finding container 5167d01538105dc526d6073ab696a5afaa6c9a4fda925b4d02b71df007726543: Status 404 returned error can't find the container with id 5167d01538105dc526d6073ab696a5afaa6c9a4fda925b4d02b71df007726543 Nov 03 19:37:01.166222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:01.166106 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-57shs"] Nov 03 19:37:01.168473 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:37:01.168444 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea15765_cb00_498d_a197_4550768d770b.slice/crio-ec58ac29ff4d78e7747cd9ef4a50ebc0199ba345bce83cb29a773f56f07fe301 WatchSource:0}: Error finding container ec58ac29ff4d78e7747cd9ef4a50ebc0199ba345bce83cb29a773f56f07fe301: Status 404 returned error can't find the container with id ec58ac29ff4d78e7747cd9ef4a50ebc0199ba345bce83cb29a773f56f07fe301 Nov 03 19:37:01.610701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:01.610664 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" event={"ID":"784da5d0-b4e7-4590-a365-2be891af94c2","Type":"ContainerStarted","Data":"5167d01538105dc526d6073ab696a5afaa6c9a4fda925b4d02b71df007726543"} Nov 03 19:37:01.612167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:01.612138 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" event={"ID":"bea15765-cb00-498d-a197-4550768d770b","Type":"ContainerStarted","Data":"ec58ac29ff4d78e7747cd9ef4a50ebc0199ba345bce83cb29a773f56f07fe301"} Nov 03 19:37:04.631332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:04.631292 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" event={"ID":"784da5d0-b4e7-4590-a365-2be891af94c2","Type":"ContainerStarted","Data":"95779c7e1dec1fae40ac9bc9305f518f70c5cc68f40ad5b68a66239c4d588978"} Nov 03 19:37:04.631773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:04.631368 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:04.632884 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:04.632861 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" event={"ID":"bea15765-cb00-498d-a197-4550768d770b","Type":"ContainerStarted","Data":"a30c144e7d2f9d6f8d2c4936b8d2d9b6978f63a71cf1187d9075b52c87f6762e"} Nov 03 19:37:04.633045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:04.632908 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:04.653056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:04.653015 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" podStartSLOduration=2.170709354 podStartE2EDuration="4.653001961s" totalImagesPullingTime="2.482292607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:04.650364948 +0000 UTC m=+590.991478432" watchObservedRunningTime="2026-11-03 19:37:04.653001961 +0000 UTC m=+590.994115439" Nov 03 19:37:04.671158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:04.671116 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" podStartSLOduration=2.186785212 podStartE2EDuration="4.671101542s" totalImagesPullingTime="2.48431633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:04.668954363 +0000 UTC m=+591.010067850" watchObservedRunningTime="2026-11-03 19:37:04.671101542 +0000 UTC m=+591.012215027" Nov 03 19:37:10.556587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.556551 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc"] Nov 03 19:37:10.564723 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.564697 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.571849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.571814 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc"] Nov 03 19:37:10.659067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659039 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659117 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659149 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659265 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659324 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659369 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659459 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac123bf-79b0-466c-9546-18b94aadcbf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659499 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.659560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.659533 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrcs9\" (UniqueName: \"kubernetes.io/projected/dac123bf-79b0-466c-9546-18b94aadcbf1-kube-api-access-xrcs9\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760478 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760526 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760559 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760586 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760643 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac123bf-79b0-466c-9546-18b94aadcbf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760666 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760691 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrcs9\" (UniqueName: \"kubernetes.io/projected/dac123bf-79b0-466c-9546-18b94aadcbf1-kube-api-access-xrcs9\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760722 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.760809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760786 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.761020 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.760999 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.761075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.761039 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.761117 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.761072 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.761158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.761129 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.761578 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.761561 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac123bf-79b0-466c-9546-18b94aadcbf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.763135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.763113 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.763206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.763186 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.773810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.773782 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac123bf-79b0-466c-9546-18b94aadcbf1-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.773904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.773855 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrcs9\" (UniqueName: \"kubernetes.io/projected/dac123bf-79b0-466c-9546-18b94aadcbf1-kube-api-access-xrcs9\") pod \"maas-default-gateway-openshift-default-66db998887-jtkbc\" (UID: \"dac123bf-79b0-466c-9546-18b94aadcbf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:10.883157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:10.883086 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:11.174959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.174928 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc"] Nov 03 19:37:11.176984 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:37:11.176936 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac123bf_79b0_466c_9546_18b94aadcbf1.slice/crio-b14b6185aa104d97f679a6c398b1679617030683bc57939005852b7746af5398 WatchSource:0}: Error finding container b14b6185aa104d97f679a6c398b1679617030683bc57939005852b7746af5398: Status 404 returned error can't find the container with id b14b6185aa104d97f679a6c398b1679617030683bc57939005852b7746af5398 Nov 03 19:37:11.179390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.179359 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:11.179482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.179437 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:11.179482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.179474 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:11.669871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.669833 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" event={"ID":"dac123bf-79b0-466c-9546-18b94aadcbf1","Type":"ContainerStarted","Data":"ad3d4b0aef746520afd7703a8b9fbf232b35c298bb06b4396c48d69e68df6570"} Nov 03 19:37:11.670306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.669932 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" event={"ID":"dac123bf-79b0-466c-9546-18b94aadcbf1","Type":"ContainerStarted","Data":"b14b6185aa104d97f679a6c398b1679617030683bc57939005852b7746af5398"} Nov 03 19:37:11.701536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.701471 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podStartSLOduration=1.7014528279999999 podStartE2EDuration="1.701452828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:11.694502546 +0000 UTC m=+598.035616032" watchObservedRunningTime="2026-11-03 19:37:11.701452828 +0000 UTC m=+598.042566315" Nov 03 19:37:11.883367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:11.883335 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:12.885023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:12.884959 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 19:37:12.885676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:12.885624 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 19:37:13.883460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:13.883423 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 19:37:13.883630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:13.883480 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 19:37:14.883721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:14.883679 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 19:37:14.884148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:14.883740 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 19:37:15.640981 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:15.640941 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" Nov 03 19:37:15.641145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:15.641030 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-57shs" Nov 03 19:37:15.883905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:15.883873 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 19:37:15.884319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:15.883924 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 19:37:16.884242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:16.884208 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 19:37:16.884625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:16.884259 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 19:37:17.884392 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:17.884357 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 19:37:17.884804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:17.884414 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 19:37:19.350593 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:19.350561 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:19.351120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:19.350954 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:19.351785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:19.351766 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" Nov 03 19:37:21.275696 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:37:21.275656 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:37:25.604552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.604517 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr"] Nov 03 19:37:25.609755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.609732 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.621706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.621678 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg"] Nov 03 19:37:25.626075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626043 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.626402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626372 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626422 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626477 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626616 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626660 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626734 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765c74d-cbf2-4d02-8212-a9635bb749ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626806 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626865 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6vf\" (UniqueName: \"kubernetes.io/projected/a765c74d-cbf2-4d02-8212-a9635bb749ea-kube-api-access-xx6vf\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.626892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.626885 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.627693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.627676 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr"] Nov 03 19:37:25.636920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.636896 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg"] Nov 03 19:37:25.727410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727379 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.727592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727413 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.727592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727437 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.727592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727515 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjqk\" (UniqueName: \"kubernetes.io/projected/0693dfcf-c351-424a-99b5-8dd89f1a2985-kube-api-access-kwjqk\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.727592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727560 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.727592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727586 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6vf\" (UniqueName: \"kubernetes.io/projected/a765c74d-cbf2-4d02-8212-a9635bb749ea-kube-api-access-xx6vf\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.727903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727605 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.727903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727640 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.727903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727660 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.727903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727793 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.727903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727884 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727922 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.727954 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0693dfcf-c351-424a-99b5-8dd89f1a2985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728019 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728053 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728076 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728100 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728143 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765c74d-cbf2-4d02-8212-a9635bb749ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728173 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728291 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728385 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728451 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.728900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.728842 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765c74d-cbf2-4d02-8212-a9635bb749ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.729943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.729922 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.730235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.730220 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.743541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.743517 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765c74d-cbf2-4d02-8212-a9635bb749ea-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.743729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.743707 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6vf\" (UniqueName: \"kubernetes.io/projected/a765c74d-cbf2-4d02-8212-a9635bb749ea-kube-api-access-xx6vf\") pod \"maas-default-gateway-openshift-default-66db998887-flqfr\" (UID: \"a765c74d-cbf2-4d02-8212-a9635bb749ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.829246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829214 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829248 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829265 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0693dfcf-c351-424a-99b5-8dd89f1a2985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829284 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829319 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829565 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829600 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829628 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829665 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjqk\" (UniqueName: \"kubernetes.io/projected/0693dfcf-c351-424a-99b5-8dd89f1a2985-kube-api-access-kwjqk\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829682 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829863 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.829957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.829895 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.830146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.830011 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.830222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.830195 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0693dfcf-c351-424a-99b5-8dd89f1a2985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.831805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.831784 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.831805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.831854 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.845536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.845513 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0693dfcf-c351-424a-99b5-8dd89f1a2985-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.845842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.845821 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjqk\" (UniqueName: \"kubernetes.io/projected/0693dfcf-c351-424a-99b5-8dd89f1a2985-kube-api-access-kwjqk\") pod \"maas-default-gateway-openshift-default-66db998887-fw2bg\" (UID: \"0693dfcf-c351-424a-99b5-8dd89f1a2985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:25.930533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.930457 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:25.946445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:25.946422 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:26.215924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.215900 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr"] Nov 03 19:37:26.217720 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:37:26.217696 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda765c74d_cbf2_4d02_8212_a9635bb749ea.slice/crio-dbda93e30a2510cd3c5d0af8b7d8e0a291d3cce4c04fa1c5cbb541c8cd36532a WatchSource:0}: Error finding container dbda93e30a2510cd3c5d0af8b7d8e0a291d3cce4c04fa1c5cbb541c8cd36532a: Status 404 returned error can't find the container with id dbda93e30a2510cd3c5d0af8b7d8e0a291d3cce4c04fa1c5cbb541c8cd36532a Nov 03 19:37:26.220160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.220130 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:26.220226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.220194 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:26.220259 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.220224 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:26.239660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.239637 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg"] Nov 03 19:37:26.241434 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:37:26.241409 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0693dfcf_c351_424a_99b5_8dd89f1a2985.slice/crio-86b863e936771b06cd5676b92c5ba6d9d91b64385e29f57942a041ec52402f64 WatchSource:0}: Error finding container 86b863e936771b06cd5676b92c5ba6d9d91b64385e29f57942a041ec52402f64: Status 404 returned error can't find the container with id 86b863e936771b06cd5676b92c5ba6d9d91b64385e29f57942a041ec52402f64 Nov 03 19:37:26.243593 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.243562 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:26.243671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.243625 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:26.243671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.243653 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:26.777951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.777918 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" event={"ID":"0693dfcf-c351-424a-99b5-8dd89f1a2985","Type":"ContainerStarted","Data":"2d40095b98c0e0344c905441f307fd88d9ac20fc1afbb27875ccb95fddbd2c3d"} Nov 03 19:37:26.778358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.777987 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" event={"ID":"0693dfcf-c351-424a-99b5-8dd89f1a2985","Type":"ContainerStarted","Data":"86b863e936771b06cd5676b92c5ba6d9d91b64385e29f57942a041ec52402f64"} Nov 03 19:37:26.779780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.779755 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" event={"ID":"a765c74d-cbf2-4d02-8212-a9635bb749ea","Type":"ContainerStarted","Data":"90b5829fa43d0cee7012bf26d066536dd6538ec45af6d9ed0370546abcf60377"} Nov 03 19:37:26.779902 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.779796 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" event={"ID":"a765c74d-cbf2-4d02-8212-a9635bb749ea","Type":"ContainerStarted","Data":"dbda93e30a2510cd3c5d0af8b7d8e0a291d3cce4c04fa1c5cbb541c8cd36532a"} Nov 03 19:37:26.851178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.851091 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podStartSLOduration=1.851072246 podStartE2EDuration="1.851072246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:26.848256517 +0000 UTC m=+613.189370002" watchObservedRunningTime="2026-11-03 19:37:26.851072246 +0000 UTC m=+613.192185734" Nov 03 19:37:26.856241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.856192 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podStartSLOduration=1.856177222 podStartE2EDuration="1.856177222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:26.821380987 +0000 UTC m=+613.162494484" watchObservedRunningTime="2026-11-03 19:37:26.856177222 +0000 UTC m=+613.197290707" Nov 03 19:37:26.935200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.935161 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:26.946998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:26.946939 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:27.936036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:27.935992 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 19:37:27.936538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:27.936495 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 19:37:27.948262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:27.948235 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 19:37:27.948425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:27.948405 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 19:37:28.933997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:28.931419 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 19:37:28.933997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:28.931483 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 19:37:28.947217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:28.947175 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 19:37:28.947630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:28.947251 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 19:37:29.932014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:29.931495 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 19:37:29.932014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:29.931572 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 19:37:29.947533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:29.947430 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 19:37:29.947533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:29.947480 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 19:37:30.932871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:30.932383 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 19:37:30.932871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:30.932445 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 19:37:30.947342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:30.947255 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 19:37:30.947342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:30.947299 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 19:37:31.936125 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:31.936084 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 19:37:31.936837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:31.936801 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 19:37:31.946809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:31.946782 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 19:37:31.946928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:31.946850 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 19:37:32.931537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:32.931489 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 19:37:32.931741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:32.931553 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 19:37:32.947563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:32.947527 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 19:37:32.947899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:32.947597 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 19:37:34.529996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:34.529281 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:34.530473 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:34.530384 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:34.531909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:34.531859 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" Nov 03 19:37:34.571138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:34.571108 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:34.571535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:34.571513 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:34.572370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:34.572351 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" Nov 03 19:37:51.276374 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:37:51.276337 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:37:54.483296 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.483263 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh"] Nov 03 19:37:54.490738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.490716 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.508596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.508571 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-435bd8-openshift-default-dockercfg-zznl2\"" Nov 03 19:37:54.511788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.511761 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh"] Nov 03 19:37:54.573715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.573678 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.573715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.573720 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.573933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.573819 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.573933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.573838 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.573933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.573882 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.574082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.573996 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.574082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.574044 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.574188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.574136 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.574188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.574172 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wv6x\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.672232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.672198 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc"] Nov 03 19:37:54.675514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675487 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.675647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675556 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.675647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675593 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.675647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675619 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wv6x\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.675801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675686 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.675881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675860 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.675938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675900 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675938 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675952 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.675988 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.676070 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.676302 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.676421 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.676495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.676468 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.676558 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.676523 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.678202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.678175 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.678565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.678545 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.681394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.681373 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2c3fc4-openshift-default-dockercfg-f8v5n\"" Nov 03 19:37:54.691563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.691531 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc"] Nov 03 19:37:54.692230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.692204 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.694772 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.694750 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wv6x\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.777598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777517 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777600 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzf4\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777642 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777724 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777797 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777819 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777837 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.777878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.777862 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.778114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.778028 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.804635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.804604 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:54.881589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881405 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881471 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881529 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881562 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881591 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881629 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881670 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881729 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.881845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.881804 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzf4\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.883239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.883209 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.883399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.883374 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.883665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.883643 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.884297 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.884096 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.884297 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.884100 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.885592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.885565 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.887003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.886376 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.898302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.898277 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzf4\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.899055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.899026 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:54.954112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.954082 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4"] Nov 03 19:37:54.959271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.959250 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:54.988735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:54.988703 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d8a4ea-openshift-default-dockercfg-wkq5q\"" Nov 03 19:37:55.006555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.006509 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4"] Nov 03 19:37:55.019787 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.019757 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:55.090705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.090660 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.090864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.090760 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.090864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.090820 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.091062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.091033 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.091136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.091081 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzcd\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.091204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.091182 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.091261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.091234 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.091349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.091330 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.091407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.091371 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.099311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.099282 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh"] Nov 03 19:37:55.104587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.104552 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.104682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.104643 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.104726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.104688 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192106 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192216 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192252 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzcd\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192292 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192356 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192420 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192462 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192522 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.192999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.192558 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.194876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.193937 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.194876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.194099 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.195284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.195021 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.195348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.195319 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.195525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.195442 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.198294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.198272 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.198870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.198827 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.207815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.207779 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.209860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.209839 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzcd\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.284572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.284493 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:55.304226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.304196 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc"] Nov 03 19:37:55.305266 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:37:55.305204 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388328ea_4202_411e_baae_80d3c6e875a3.slice/crio-e49b3b209ade411180ec3b9dd9b28a9cd1676fca2733b9ba925aae20ba4863e5 WatchSource:0}: Error finding container e49b3b209ade411180ec3b9dd9b28a9cd1676fca2733b9ba925aae20ba4863e5: Status 404 returned error can't find the container with id e49b3b209ade411180ec3b9dd9b28a9cd1676fca2733b9ba925aae20ba4863e5 Nov 03 19:37:55.308190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.308152 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.308284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.308236 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.308324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.308281 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.590392 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.590365 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4"] Nov 03 19:37:55.591793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.591763 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.591918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.591836 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.591918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.591886 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:37:55.988362 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.988322 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" event={"ID":"602cc334-6a9b-41c0-8ac3-7779884a6de2","Type":"ContainerStarted","Data":"aa25e985b942629694fec8becdeca0521b8500e64c44d02dedb36b35add4f8ab"} Nov 03 19:37:55.988560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.988391 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" event={"ID":"602cc334-6a9b-41c0-8ac3-7779884a6de2","Type":"ContainerStarted","Data":"df58f374d87340bfbfa05fb700742121f09ed93096a53d40b099a1e45dea0e6d"} Nov 03 19:37:55.990022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.989994 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" event={"ID":"2a56a2fb-fc39-4103-b90e-adc6c07f4907","Type":"ContainerStarted","Data":"f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b"} Nov 03 19:37:55.990169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.990039 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" event={"ID":"2a56a2fb-fc39-4103-b90e-adc6c07f4907","Type":"ContainerStarted","Data":"7f17eb9538ec06310bae3359b5ebbd7c3e542e3b47fd93693b7a641a5d26133d"} Nov 03 19:37:55.991994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.991941 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" event={"ID":"388328ea-4202-411e-baae-80d3c6e875a3","Type":"ContainerStarted","Data":"502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3"} Nov 03 19:37:55.992090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:55.992023 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" event={"ID":"388328ea-4202-411e-baae-80d3c6e875a3","Type":"ContainerStarted","Data":"e49b3b209ade411180ec3b9dd9b28a9cd1676fca2733b9ba925aae20ba4863e5"} Nov 03 19:37:56.014898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.014850 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podStartSLOduration=2.01482696 podStartE2EDuration="2.01482696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:56.012890278 +0000 UTC m=+642.354003762" watchObservedRunningTime="2026-11-03 19:37:56.01482696 +0000 UTC m=+642.355940444" Nov 03 19:37:56.020107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.020084 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:56.025664 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.025643 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:56.042831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.042785 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podStartSLOduration=2.042772777 podStartE2EDuration="2.042772777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:56.040638888 +0000 UTC m=+642.381752376" watchObservedRunningTime="2026-11-03 19:37:56.042772777 +0000 UTC m=+642.383886256" Nov 03 19:37:56.068593 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.068546 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podStartSLOduration=2.068533645 podStartE2EDuration="2.068533645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:56.065594767 +0000 UTC m=+642.406708253" watchObservedRunningTime="2026-11-03 19:37:56.068533645 +0000 UTC m=+642.409647129" Nov 03 19:37:56.284901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.284821 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:56.290707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.290682 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:56.805509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.805468 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:56.811449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.811423 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:56.997267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.997235 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:56.997439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.997293 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:56.997439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.997316 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:56.998513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.998488 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:37:56.998674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.998547 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:37:56.998744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:56.998713 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:37:59.507757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.507726 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48"] Nov 03 19:37:59.511742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.511721 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.520814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.520791 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-05e217\"" Nov 03 19:37:59.528683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.528653 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48"] Nov 03 19:37:59.538963 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.538933 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z"] Nov 03 19:37:59.542988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.542950 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.548820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.548783 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-05e217-dockercfg-2lnwd\"" Nov 03 19:37:59.564329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.564299 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z"] Nov 03 19:37:59.655007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.647218 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdgtq\" (UniqueName: \"kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq\") pod \"payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.655007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.647341 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.655007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.647408 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hct\" (UniqueName: \"kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct\") pod \"payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.655007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.647510 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.748528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.748495 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.748724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.748548 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hct\" (UniqueName: \"kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct\") pod \"payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.748724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.748607 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.748724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.748634 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgtq\" (UniqueName: \"kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq\") pod \"payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.749376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.749355 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.749420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.749394 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.766015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.765927 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hct\" (UniqueName: \"kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct\") pod \"payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:37:59.766416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.766391 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdgtq\" (UniqueName: \"kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq\") pod \"payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.828180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.828144 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:37:59.863344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:37:59.863314 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:38:00.109606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.109580 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48"] Nov 03 19:38:00.111329 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:00.111303 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5540ea_5820_49ef_918f_655bcffeba0c.slice/crio-aca6286b0e1e188afa1a8886eac0ed70c325adc5f0f3d69348d99d7f2b12128e WatchSource:0}: Error finding container aca6286b0e1e188afa1a8886eac0ed70c325adc5f0f3d69348d99d7f2b12128e: Status 404 returned error can't find the container with id aca6286b0e1e188afa1a8886eac0ed70c325adc5f0f3d69348d99d7f2b12128e Nov 03 19:38:00.167509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.167250 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z"] Nov 03 19:38:00.169058 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:00.169028 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6423373_89fc_451b_8b22_26e614285ad7.slice/crio-94cf5eee04803a6ad519c0e0ce8690e50366c6d97a7db5007008f713038a0a28 WatchSource:0}: Error finding container 94cf5eee04803a6ad519c0e0ce8690e50366c6d97a7db5007008f713038a0a28: Status 404 returned error can't find the container with id 94cf5eee04803a6ad519c0e0ce8690e50366c6d97a7db5007008f713038a0a28 Nov 03 19:38:00.390746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.390667 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69"] Nov 03 19:38:00.394526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.394508 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.417232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.417193 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69"] Nov 03 19:38:00.473279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.460714 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96k9\" (UniqueName: \"kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9\") pod \"payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.473279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.460794 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.473279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.461155 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-036184-dockercfg-tnp56\"" Nov 03 19:38:00.473279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.461234 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-036184\"" Nov 03 19:38:00.562393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.562352 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96k9\" (UniqueName: \"kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9\") pod \"payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.562877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.562418 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.563272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.563246 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.574903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.574868 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96k9\" (UniqueName: \"kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9\") pod \"payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:00.708475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:00.708390 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:01.017580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.017548 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69"] Nov 03 19:38:01.022357 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:01.022322 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9359b24e_557f_428e_8205_5c9035c3f76e.slice/crio-b4f7ba6b8b54ce80dc2cc026292f2afc99b1302593459333313717f251f54537 WatchSource:0}: Error finding container b4f7ba6b8b54ce80dc2cc026292f2afc99b1302593459333313717f251f54537: Status 404 returned error can't find the container with id b4f7ba6b8b54ce80dc2cc026292f2afc99b1302593459333313717f251f54537 Nov 03 19:38:01.022654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.022625 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" event={"ID":"fb5540ea-5820-49ef-918f-655bcffeba0c","Type":"ContainerStarted","Data":"d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e"} Nov 03 19:38:01.022768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.022683 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" event={"ID":"fb5540ea-5820-49ef-918f-655bcffeba0c","Type":"ContainerStarted","Data":"aca6286b0e1e188afa1a8886eac0ed70c325adc5f0f3d69348d99d7f2b12128e"} Nov 03 19:38:01.022768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.022719 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:38:01.024422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.024387 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" event={"ID":"e6423373-89fc-451b-8b22-26e614285ad7","Type":"ContainerStarted","Data":"89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08"} Nov 03 19:38:01.024507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.024439 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" event={"ID":"e6423373-89fc-451b-8b22-26e614285ad7","Type":"ContainerStarted","Data":"94cf5eee04803a6ad519c0e0ce8690e50366c6d97a7db5007008f713038a0a28"} Nov 03 19:38:01.024507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.024467 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:38:01.049325 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.049275 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" podStartSLOduration=2.049255629 podStartE2EDuration="2.049255629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:01.044520522 +0000 UTC m=+647.385634007" watchObservedRunningTime="2026-11-03 19:38:01.049255629 +0000 UTC m=+647.390369114" Nov 03 19:38:01.066109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.066051 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" podStartSLOduration=2.06603277 podStartE2EDuration="2.06603277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:01.063934219 +0000 UTC m=+647.405047705" watchObservedRunningTime="2026-11-03 19:38:01.06603277 +0000 UTC m=+647.407146256" Nov 03 19:38:01.646021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.645964 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr"] Nov 03 19:38:01.650201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.650172 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.655455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.655188 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-2c3fc4\"" Nov 03 19:38:01.659905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.659719 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr"] Nov 03 19:38:01.798724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.798676 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.798990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.798956 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzq94\" (UniqueName: \"kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94\") pod \"payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.900606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.900510 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.900606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.900552 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzq94\" (UniqueName: \"kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94\") pod \"payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.901424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.901393 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.914420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.914391 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzq94\" (UniqueName: \"kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94\") pod \"payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:01.968760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:01.968667 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:02.042481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.041471 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" event={"ID":"9359b24e-557f-428e-8205-5c9035c3f76e","Type":"ContainerStarted","Data":"a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5"} Nov 03 19:38:02.042481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.041520 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" event={"ID":"9359b24e-557f-428e-8205-5c9035c3f76e","Type":"ContainerStarted","Data":"b4f7ba6b8b54ce80dc2cc026292f2afc99b1302593459333313717f251f54537"} Nov 03 19:38:02.042481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.041938 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:02.062284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.062232 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" podStartSLOduration=2.062216236 podStartE2EDuration="2.062216236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:02.061404391 +0000 UTC m=+648.402517876" watchObservedRunningTime="2026-11-03 19:38:02.062216236 +0000 UTC m=+648.403329764" Nov 03 19:38:02.270267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.270232 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr"] Nov 03 19:38:02.423259 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.423226 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n"] Nov 03 19:38:02.427166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.427137 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.436985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.435650 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-435bd8\"" Nov 03 19:38:02.437793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.437773 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-435bd8-dockercfg-5bdq8\"" Nov 03 19:38:02.439787 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.439758 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n"] Nov 03 19:38:02.450219 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.450187 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp"] Nov 03 19:38:02.454242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.454220 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.469309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.469281 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp"] Nov 03 19:38:02.612709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.612610 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.619118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.618076 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qr6w\" (UniqueName: \"kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w\") pod \"payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.619118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.618169 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwh9m\" (UniqueName: \"kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m\") pod \"payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.619118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.618220 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.719598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.719563 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.720062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.719698 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.720062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.719742 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qr6w\" (UniqueName: \"kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w\") pod \"payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.720062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.719791 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwh9m\" (UniqueName: \"kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m\") pod \"payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.720554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.720524 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.720674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.720588 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.732538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.732512 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwh9m\" (UniqueName: \"kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m\") pod \"payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.733284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.733262 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qr6w\" (UniqueName: \"kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w\") pod \"payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.747136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.747104 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:02.774027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.773995 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:02.888062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.884990 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv"] Nov 03 19:38:02.893556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.893530 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:02.902943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.902262 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d8a4ea\"" Nov 03 19:38:02.902943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.902870 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv"] Nov 03 19:38:02.930857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.930621 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8dwf\" (UniqueName: \"kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf\") pod \"payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:02.930857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:02.930714 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:03.031744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.031709 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dwf\" (UniqueName: \"kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf\") pod \"payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:03.031958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.031874 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:03.032602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.032583 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:03.041797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.040158 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n"] Nov 03 19:38:03.046621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.046344 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8dwf\" (UniqueName: \"kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf\") pod \"payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:03.058307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.056834 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" event={"ID":"f24de34e-fb61-4398-b65f-7d2c552a91a6","Type":"ContainerStarted","Data":"2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab"} Nov 03 19:38:03.058307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.056895 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" event={"ID":"f24de34e-fb61-4398-b65f-7d2c552a91a6","Type":"ContainerStarted","Data":"e3a8a55e64258a14c0fb5f27fa02c4dc2098efe0ca108e0f59400b0580ea1f19"} Nov 03 19:38:03.058307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.056942 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:03.059160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.059110 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" event={"ID":"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5","Type":"ContainerStarted","Data":"753322f563a3249abbdd82edb9e86f1a426842b9ecbe75a0e7f63d59a816697d"} Nov 03 19:38:03.079430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.079373 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" podStartSLOduration=2.079353171 podStartE2EDuration="2.079353171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:03.077393676 +0000 UTC m=+649.418507157" watchObservedRunningTime="2026-11-03 19:38:03.079353171 +0000 UTC m=+649.420466657" Nov 03 19:38:03.082774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.082747 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp"] Nov 03 19:38:03.086301 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:03.086279 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610b32be_07bd_4b82_89cb_ad4b63638981.slice/crio-3974733b717e333b8bfb4780766a50790db62b3df5f92a49f3568e558ab9b4f7 WatchSource:0}: Error finding container 3974733b717e333b8bfb4780766a50790db62b3df5f92a49f3568e558ab9b4f7: Status 404 returned error can't find the container with id 3974733b717e333b8bfb4780766a50790db62b3df5f92a49f3568e558ab9b4f7 Nov 03 19:38:03.227346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.227300 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:03.548815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.548728 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv"] Nov 03 19:38:03.555891 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:03.555848 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6204480_8a38_4886_8b54_fc88eb69902e.slice/crio-a3d2b9a2306e0aa920e4768953d555b9b76d77d65db1479f47b45f92e6173588 WatchSource:0}: Error finding container a3d2b9a2306e0aa920e4768953d555b9b76d77d65db1479f47b45f92e6173588: Status 404 returned error can't find the container with id a3d2b9a2306e0aa920e4768953d555b9b76d77d65db1479f47b45f92e6173588 Nov 03 19:38:03.633516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.633487 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n"] Nov 03 19:38:03.637440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.637420 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.640952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.640925 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwj6m\" (UniqueName: \"kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m\") pod \"payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.641098 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.640992 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.644988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.644940 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-c5f6cb\"" Nov 03 19:38:03.649064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.649003 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n"] Nov 03 19:38:03.660193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.660159 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-c5f6cb-dockercfg-drpcd\"" Nov 03 19:38:03.683759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.683721 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k"] Nov 03 19:38:03.688608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.688586 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.698046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.698018 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k"] Nov 03 19:38:03.742603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.742562 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.743246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.742672 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.743246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.742737 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qtz\" (UniqueName: \"kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz\") pod \"payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.743246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.742875 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwj6m\" (UniqueName: \"kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m\") pod \"payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.744170 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.744143 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.769698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.769618 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwj6m\" (UniqueName: \"kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m\") pod \"payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:03.844155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.844075 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.844155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.844124 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qtz\" (UniqueName: \"kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz\") pod \"payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.844766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.844746 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.857477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.857442 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qtz\" (UniqueName: \"kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz\") pod \"payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:03.955996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:03.955941 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:04.011010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.010962 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.072219 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" event={"ID":"610b32be-07bd-4b82-89cb-ad4b63638981","Type":"ContainerStarted","Data":"6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f"} Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.072274 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" event={"ID":"610b32be-07bd-4b82-89cb-ad4b63638981","Type":"ContainerStarted","Data":"3974733b717e333b8bfb4780766a50790db62b3df5f92a49f3568e558ab9b4f7"} Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.072601 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.074348 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" event={"ID":"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5","Type":"ContainerStarted","Data":"4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7"} Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.074475 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.076480 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" event={"ID":"c6204480-8a38-4886-8b54-fc88eb69902e","Type":"ContainerStarted","Data":"8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6"} Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.076527 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" event={"ID":"c6204480-8a38-4886-8b54-fc88eb69902e","Type":"ContainerStarted","Data":"a3d2b9a2306e0aa920e4768953d555b9b76d77d65db1479f47b45f92e6173588"} Nov 03 19:38:04.076714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.076576 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:04.103682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.095177 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" podStartSLOduration=2.095161144 podStartE2EDuration="2.095161144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:04.091916296 +0000 UTC m=+650.433029783" watchObservedRunningTime="2026-11-03 19:38:04.095161144 +0000 UTC m=+650.436274629" Nov 03 19:38:04.114658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.114598 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" podStartSLOduration=2.114581028 podStartE2EDuration="2.114581028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:04.110961996 +0000 UTC m=+650.452075474" watchObservedRunningTime="2026-11-03 19:38:04.114581028 +0000 UTC m=+650.455694515" Nov 03 19:38:04.135462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.135412 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" podStartSLOduration=2.135397913 podStartE2EDuration="2.135397913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:04.132108642 +0000 UTC m=+650.473222127" watchObservedRunningTime="2026-11-03 19:38:04.135397913 +0000 UTC m=+650.476511399" Nov 03 19:38:04.240236 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:04.240187 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9702048_263e_47da_9fe4_51b24787a1d4.slice/crio-414bf11780d7627fa459534cdbf620c71fefc7caef9f612d889b9b67b410a249 WatchSource:0}: Error finding container 414bf11780d7627fa459534cdbf620c71fefc7caef9f612d889b9b67b410a249: Status 404 returned error can't find the container with id 414bf11780d7627fa459534cdbf620c71fefc7caef9f612d889b9b67b410a249 Nov 03 19:38:04.240400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.240377 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n"] Nov 03 19:38:04.309138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:04.309116 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k"] Nov 03 19:38:04.311208 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:04.311178 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f81783_cd80_4436_bc8e_d9870342432e.slice/crio-5a12b8de281ff57ef275771d5e1f81171b6eb822df82e40136ad25e79e2a5a6f WatchSource:0}: Error finding container 5a12b8de281ff57ef275771d5e1f81171b6eb822df82e40136ad25e79e2a5a6f: Status 404 returned error can't find the container with id 5a12b8de281ff57ef275771d5e1f81171b6eb822df82e40136ad25e79e2a5a6f Nov 03 19:38:05.091821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.091777 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" event={"ID":"d9702048-263e-47da-9fe4-51b24787a1d4","Type":"ContainerStarted","Data":"bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16"} Nov 03 19:38:05.092307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.091882 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" event={"ID":"d9702048-263e-47da-9fe4-51b24787a1d4","Type":"ContainerStarted","Data":"414bf11780d7627fa459534cdbf620c71fefc7caef9f612d889b9b67b410a249"} Nov 03 19:38:05.092307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.092247 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:05.096620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.096587 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" event={"ID":"62f81783-cd80-4436-bc8e-d9870342432e","Type":"ContainerStarted","Data":"9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa"} Nov 03 19:38:05.096869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.096820 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" event={"ID":"62f81783-cd80-4436-bc8e-d9870342432e","Type":"ContainerStarted","Data":"5a12b8de281ff57ef275771d5e1f81171b6eb822df82e40136ad25e79e2a5a6f"} Nov 03 19:38:05.119333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.119271 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" podStartSLOduration=2.1192524600000002 podStartE2EDuration="2.11925246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:05.116425837 +0000 UTC m=+651.457539322" watchObservedRunningTime="2026-11-03 19:38:05.11925246 +0000 UTC m=+651.460365945" Nov 03 19:38:05.142644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:05.142583 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" podStartSLOduration=2.142564875 podStartE2EDuration="2.142564875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:05.13990055 +0000 UTC m=+651.481014035" watchObservedRunningTime="2026-11-03 19:38:05.142564875 +0000 UTC m=+651.483678361" Nov 03 19:38:06.102746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:06.102705 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:12.048997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:12.048756 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:38:12.049501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:12.049280 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:38:13.045995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:13.045139 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:38:13.045995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:13.045202 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:38:14.079753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:14.079720 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:38:14.080832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:14.080811 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:38:15.143213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:15.142165 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:38:15.152017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:15.151469 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:38:15.152411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:15.152388 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:38:16.113147 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:16.112152 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:38:17.124213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:17.124183 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:38:19.352157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:19.352119 5233 generic.go:356] "Generic (PLEG): container finished" podID="388328ea-4202-411e-baae-80d3c6e875a3" containerID="502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3" exitCode=0 Nov 03 19:38:19.352621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:19.352197 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" event={"ID":"388328ea-4202-411e-baae-80d3c6e875a3","Type":"ContainerDied","Data":"502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3"} Nov 03 19:38:19.352697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:19.352649 5233 scope.go:122] "RemoveContainer" containerID="502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3" Nov 03 19:38:19.354291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:19.354258 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:19.354379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:19.354350 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:19.354418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:19.354397 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:20.365687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:20.364635 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" event={"ID":"388328ea-4202-411e-baae-80d3c6e875a3","Type":"ContainerStarted","Data":"c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52"} Nov 03 19:38:21.020415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.020381 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:38:21.020797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.020701 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:38:21.021097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.021064 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 19:38:21.281152 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:38:21.281061 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:38:21.374785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.374750 5233 generic.go:356] "Generic (PLEG): container finished" podID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerID="aa25e985b942629694fec8becdeca0521b8500e64c44d02dedb36b35add4f8ab" exitCode=0 Nov 03 19:38:21.375365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.374798 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" event={"ID":"602cc334-6a9b-41c0-8ac3-7779884a6de2","Type":"ContainerDied","Data":"aa25e985b942629694fec8becdeca0521b8500e64c44d02dedb36b35add4f8ab"} Nov 03 19:38:21.375811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.375788 5233 scope.go:122] "RemoveContainer" containerID="aa25e985b942629694fec8becdeca0521b8500e64c44d02dedb36b35add4f8ab" Nov 03 19:38:21.377215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.377186 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:21.377272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.377259 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:21.377307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:21.377298 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:22.024457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.024414 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:38:22.024671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.024480 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 19:38:22.383839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.383752 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" event={"ID":"602cc334-6a9b-41c0-8ac3-7779884a6de2","Type":"ContainerStarted","Data":"3f4cdeaf4e2ad298769772f47d15d520abef9fbf58bf3ecd6cf57603a7eff2c0"} Nov 03 19:38:22.386806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.386773 5233 generic.go:356] "Generic (PLEG): container finished" podID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerID="f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b" exitCode=0 Nov 03 19:38:22.387103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.387078 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" event={"ID":"2a56a2fb-fc39-4103-b90e-adc6c07f4907","Type":"ContainerDied","Data":"f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b"} Nov 03 19:38:22.387646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.387625 5233 scope.go:122] "RemoveContainer" containerID="f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b" Nov 03 19:38:22.389896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.389864 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:22.390110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.390086 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:22.390265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:22.390246 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:38:23.022008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.020386 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:38:23.022008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.020451 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 19:38:23.289055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.288269 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:38:23.312991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.297870 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:38:23.312991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.297950 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 19:38:23.421046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.420127 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" event={"ID":"2a56a2fb-fc39-4103-b90e-adc6c07f4907","Type":"ContainerStarted","Data":"d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d"} Nov 03 19:38:23.929759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.929723 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:38:23.930137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:23.930107 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 19:38:24.043875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.043842 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:38:24.048210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.044564 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:38:24.048440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.048410 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 19:38:24.288441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.288401 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:38:24.288745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.288714 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 19:38:24.805995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.805701 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:38:24.805995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:24.805762 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 19:38:25.021347 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:25.021308 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:38:25.021623 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:25.021593 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 19:38:25.287108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:25.287074 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:38:25.287365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:25.287337 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 19:38:25.818428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:25.818388 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:38:25.819079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:25.819043 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 19:38:26.046995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.045236 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:38:26.046995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.046678 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:38:26.060012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.058362 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:38:26.335007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.296222 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:38:26.335340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.335298 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 19:38:26.756610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.756578 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv"] Nov 03 19:38:26.760875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.760850 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.769995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.769953 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"default-dockercfg-lhd72\"" Nov 03 19:38:26.771024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.770997 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"kube-root-ca.crt\"" Nov 03 19:38:26.771127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.771054 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"openshift-service-ca.crt\"" Nov 03 19:38:26.771197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.771151 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2c3fc4\"/\"simulator-w1-2c3fc4-kserve-self-signed-certs\"" Nov 03 19:38:26.775806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.775785 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv"] Nov 03 19:38:26.829993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.811521 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:38:26.829993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.811583 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 19:38:26.855453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.855418 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvrw\" (UniqueName: \"kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.855687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.855670 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.855781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.855768 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.855920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.855902 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.856024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.855956 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.856091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.856059 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.957304 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.957479 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvrw\" (UniqueName: \"kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.957556 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.957582 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.957625 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.957678 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.961998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.958879 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.961998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.959725 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.961998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.960845 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.964542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.964488 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.970075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.969949 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:26.973146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:26.973125 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvrw\" (UniqueName: \"kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw\") pod \"simulator-w1-2c3fc4-kserve-677f87c447-tlxrv\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:27.011670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.010077 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:38:27.011670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.010992 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:38:27.077018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.076953 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:27.291060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.289327 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:38:27.291060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.289385 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 19:38:27.464592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.464551 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv"] Nov 03 19:38:27.466300 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:27.466259 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab25e7c7_0b58_48ec_be0e_51e8329c040f.slice/crio-026ff76129a5c4d362d852a8d0f0ef6cf50455f3f5ba350645768bd220e7ecae WatchSource:0}: Error finding container 026ff76129a5c4d362d852a8d0f0ef6cf50455f3f5ba350645768bd220e7ecae: Status 404 returned error can't find the container with id 026ff76129a5c4d362d852a8d0f0ef6cf50455f3f5ba350645768bd220e7ecae Nov 03 19:38:27.571198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.571170 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" event={"ID":"ab25e7c7-0b58-48ec-be0e-51e8329c040f","Type":"ContainerStarted","Data":"026ff76129a5c4d362d852a8d0f0ef6cf50455f3f5ba350645768bd220e7ecae"} Nov 03 19:38:27.807751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.807716 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:38:27.808058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:27.808031 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 19:38:28.311351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:28.311305 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:38:28.313639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:28.313613 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:38:28.579262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:28.579173 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" event={"ID":"ab25e7c7-0b58-48ec-be0e-51e8329c040f","Type":"ContainerStarted","Data":"2cae899439c1960be425c0df9485d1d16862d554441125de606cc3dd5d817b7b"} Nov 03 19:38:28.612022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:28.611316 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podStartSLOduration=2.611296051 podStartE2EDuration="2.611296051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:28.607401427 +0000 UTC m=+674.948514914" watchObservedRunningTime="2026-11-03 19:38:28.611296051 +0000 UTC m=+674.952409537" Nov 03 19:38:28.809996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:28.807356 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:38:28.809996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:28.807417 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 19:38:29.078028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:29.077998 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:29.809661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:29.809633 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:38:29.810771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:29.810749 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:38:33.590907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.590858 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446"] Nov 03 19:38:33.596509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.596485 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.601470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.601447 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446"] Nov 03 19:38:33.604861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.604833 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"openshift-service-ca.crt\"" Nov 03 19:38:33.605056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.604847 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"kube-root-ca.crt\"" Nov 03 19:38:33.605056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.604917 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"default-dockercfg-9bbw9\"" Nov 03 19:38:33.605218 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.604919 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"simulator-w6-c5f6cb-kserve-self-signed-certs\"" Nov 03 19:38:33.777996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.770611 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.777996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.770669 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.777996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.770700 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.777996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.770765 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmsg\" (UniqueName: \"kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.777996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.770882 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.777996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.770952 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872108 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmsg\" (UniqueName: \"kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872205 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872250 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872310 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872332 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872354 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872728 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872770 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.872875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.872826 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.874564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.874542 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.874680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.874663 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.884831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.884811 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmsg\" (UniqueName: \"kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg\") pod \"simulator-w6-c5f6cb-kserve-f59897f86-9j446\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:33.912675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:33.912653 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:34.200077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:34.200052 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446"] Nov 03 19:38:34.201695 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:34.201668 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb328621b_8a79_4fe3_8c9e_2ef238b93a14.slice/crio-093bac7f2223601e168efc439249d8997218272a77bfb46c4df38a0ae40222a9 WatchSource:0}: Error finding container 093bac7f2223601e168efc439249d8997218272a77bfb46c4df38a0ae40222a9: Status 404 returned error can't find the container with id 093bac7f2223601e168efc439249d8997218272a77bfb46c4df38a0ae40222a9 Nov 03 19:38:34.624924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:34.624890 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" event={"ID":"b328621b-8a79-4fe3-8c9e-2ef238b93a14","Type":"ContainerStarted","Data":"e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649"} Nov 03 19:38:34.625336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:34.624941 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" event={"ID":"b328621b-8a79-4fe3-8c9e-2ef238b93a14","Type":"ContainerStarted","Data":"093bac7f2223601e168efc439249d8997218272a77bfb46c4df38a0ae40222a9"} Nov 03 19:38:34.646395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:34.646345 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podStartSLOduration=1.646330962 podStartE2EDuration="1.646330962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:34.644653923 +0000 UTC m=+680.985767409" watchObservedRunningTime="2026-11-03 19:38:34.646330962 +0000 UTC m=+680.987444449" Nov 03 19:38:34.913769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:34.913692 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:37.077607 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:37.077570 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:37.092645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:37.092614 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:37.660039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:37.660001 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:38:43.913365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:43.913328 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:43.926094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:43.926065 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:44.704683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:44.704646 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:38:51.282310 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:38:51.282271 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:38:51.534748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.534663 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt"] Nov 03 19:38:51.549761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.549730 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt"] Nov 03 19:38:51.549930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.549836 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.560320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.560226 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.560511 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.560453 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.567013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.566894 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"premium-w2-435bd8-kserve-self-signed-certs\"" Nov 03 19:38:51.567152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.567056 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"default-dockercfg-nt8gv\"" Nov 03 19:38:51.693439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.693397 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.693773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.693748 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.693920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.693896 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.694078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.693951 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.694078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.694033 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qd2h\" (UniqueName: \"kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.694256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.694235 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.795771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.795675 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.795771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.795732 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.795775 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.795835 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qd2h\" (UniqueName: \"kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.795938 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.796003 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.796143 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.796194 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.796681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.796658 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.798321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.798297 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.798725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.798704 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.808861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.808836 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qd2h\" (UniqueName: \"kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h\") pod \"premium-w2-435bd8-kserve-697d5d5c46-478tt\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:51.874040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:51.874011 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:52.172730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:52.172698 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt"] Nov 03 19:38:52.174231 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:52.174197 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd98abe45_77ff_413a_8ea8_c88f9a7a26cd.slice/crio-e7056565330b2cccb2f16544de50b8bfc9b1c7e015d2f95b85184bdb73f6829c WatchSource:0}: Error finding container e7056565330b2cccb2f16544de50b8bfc9b1c7e015d2f95b85184bdb73f6829c: Status 404 returned error can't find the container with id e7056565330b2cccb2f16544de50b8bfc9b1c7e015d2f95b85184bdb73f6829c Nov 03 19:38:52.765027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:52.764990 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" event={"ID":"d98abe45-77ff-413a-8ea8-c88f9a7a26cd","Type":"ContainerStarted","Data":"f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99"} Nov 03 19:38:52.765441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:52.765038 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" event={"ID":"d98abe45-77ff-413a-8ea8-c88f9a7a26cd","Type":"ContainerStarted","Data":"e7056565330b2cccb2f16544de50b8bfc9b1c7e015d2f95b85184bdb73f6829c"} Nov 03 19:38:52.789191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:52.789121 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podStartSLOduration=1.7891072110000001 podStartE2EDuration="1.789107211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:52.787261871 +0000 UTC m=+699.128375356" watchObservedRunningTime="2026-11-03 19:38:52.789107211 +0000 UTC m=+699.130220695" Nov 03 19:38:52.874627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:52.874597 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:38:57.305938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.305906 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5"] Nov 03 19:38:57.309965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.309931 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.322784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.322754 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5"] Nov 03 19:38:57.323027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.322930 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"premium-w6-c5f6cb-kserve-self-signed-certs\"" Nov 03 19:38:57.482648 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.482612 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.482840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.482665 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.482840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.482687 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvg2\" (UniqueName: \"kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.482840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.482756 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.483051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.482860 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.483051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.482918 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.584743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.584642 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.584924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.584747 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.584924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.584847 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.584924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.584900 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.585132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.584935 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvg2\" (UniqueName: \"kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.585132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.584991 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.585408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.585374 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.585754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.585715 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.585754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.585752 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.587668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.587641 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.588209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.588184 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.631301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.631271 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvg2\" (UniqueName: \"kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2\") pod \"premium-w6-c5f6cb-kserve-77dffb797f-tw5c5\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:57.928487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:57.928401 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:38:58.223856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:58.223826 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5"] Nov 03 19:38:58.226195 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:38:58.226156 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod294e8459_7193_46ec_81b9_5a1e0822eb92.slice/crio-f6e2deb28cd2b322a8d0622ecc4bbb4f2b8ca21e22652b19eef2ac77b3f2328a WatchSource:0}: Error finding container f6e2deb28cd2b322a8d0622ecc4bbb4f2b8ca21e22652b19eef2ac77b3f2328a: Status 404 returned error can't find the container with id f6e2deb28cd2b322a8d0622ecc4bbb4f2b8ca21e22652b19eef2ac77b3f2328a Nov 03 19:38:58.832145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:58.832112 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" event={"ID":"294e8459-7193-46ec-81b9-5a1e0822eb92","Type":"ContainerStarted","Data":"7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4"} Nov 03 19:38:58.832544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:58.832157 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" event={"ID":"294e8459-7193-46ec-81b9-5a1e0822eb92","Type":"ContainerStarted","Data":"f6e2deb28cd2b322a8d0622ecc4bbb4f2b8ca21e22652b19eef2ac77b3f2328a"} Nov 03 19:38:58.856590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:58.856542 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podStartSLOduration=1.8565297360000002 podStartE2EDuration="1.856529736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:58.852539318 +0000 UTC m=+705.193652803" watchObservedRunningTime="2026-11-03 19:38:58.856529736 +0000 UTC m=+705.197643221" Nov 03 19:38:58.928570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:38:58.928536 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:39:01.889000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:01.882262 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:39:01.914026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:01.912603 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:39:02.879161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:02.879135 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:39:07.928566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:07.928533 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:39:07.946939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:07.946917 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:39:08.916009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:08.915959 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:39:09.344541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.344505 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2"] Nov 03 19:39:09.349738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.349719 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.358002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.357843 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-036184-openshift-default-dockercfg-xkkwv\"" Nov 03 19:39:09.360780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.360288 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2"] Nov 03 19:39:09.438590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438560 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.438796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438600 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.438796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438716 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.438796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438741 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.438796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438762 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.439053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438819 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.439053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438884 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.439053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.438947 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mt66\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.439053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.439011 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.539594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539565 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.539791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539633 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mt66\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.539791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539653 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.539791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539691 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.539791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539727 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539813 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539835 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539859 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.539896 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.540169 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.540235 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.540305 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.540512 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.540762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.540741 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.542100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.542080 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.542361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.542331 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.556576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.556554 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.556730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.556714 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mt66\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.670276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.670202 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:09.971326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.971299 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2"] Nov 03 19:39:09.972348 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:39:09.972316 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353f8791_bbf9_4b2b_97c9_9e6695a03721.slice/crio-f6e193a6c2dd1231ac2255293099be6393a28af58007923e7730f125c978a9e2 WatchSource:0}: Error finding container f6e193a6c2dd1231ac2255293099be6393a28af58007923e7730f125c978a9e2: Status 404 returned error can't find the container with id f6e193a6c2dd1231ac2255293099be6393a28af58007923e7730f125c978a9e2 Nov 03 19:39:09.974635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.974604 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:09.974734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.974664 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:09.974734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.974690 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:09.997073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:09.997050 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8"] Nov 03 19:39:10.003084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.003062 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.008106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.007822 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c5f6cb-openshift-default-dockercfg-p6txp\"" Nov 03 19:39:10.015499 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.015476 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8"] Nov 03 19:39:10.046432 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046407 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.046556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046445 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.046556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046536 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.046676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046602 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.046676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046652 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.046785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046698 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.046874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046837 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.047006 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.046989 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.047078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.047039 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vz4\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.148646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148604 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vz4\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.148834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148687 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.148834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148723 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.148834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148776 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.148834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148821 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148877 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148928 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.148956 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.149105 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.149166 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.149180 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.149474 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.149578 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.149755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.149712 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.151617 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.151592 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.151715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.151631 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.161071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.161043 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.161285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.161264 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vz4\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.332303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.332219 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:10.638063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.638038 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8"] Nov 03 19:39:10.640542 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:39:10.640513 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a8b3d6_8943_4fa5_a8c7_ed9c27b3f2fe.slice/crio-838c2a4d82b280689a0fbdadbd4accbc0f6cbb375bb9933bbc33cb4bf711f25d WatchSource:0}: Error finding container 838c2a4d82b280689a0fbdadbd4accbc0f6cbb375bb9933bbc33cb4bf711f25d: Status 404 returned error can't find the container with id 838c2a4d82b280689a0fbdadbd4accbc0f6cbb375bb9933bbc33cb4bf711f25d Nov 03 19:39:10.643026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.642990 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:10.643133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.643057 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:10.643133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.643085 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:10.932942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.932768 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" event={"ID":"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe","Type":"ContainerStarted","Data":"0275cb42d94646bd2fb47fac741975c9159f5296410e2d99f5a836d9c03a5830"} Nov 03 19:39:10.932942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.932834 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" event={"ID":"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe","Type":"ContainerStarted","Data":"838c2a4d82b280689a0fbdadbd4accbc0f6cbb375bb9933bbc33cb4bf711f25d"} Nov 03 19:39:10.935084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.935052 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" event={"ID":"353f8791-bbf9-4b2b-97c9-9e6695a03721","Type":"ContainerStarted","Data":"730976ec4e183b7307f63ea1fbd209f3a168399be2e550f2a556b26bf5c23247"} Nov 03 19:39:10.935214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.935106 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" event={"ID":"353f8791-bbf9-4b2b-97c9-9e6695a03721","Type":"ContainerStarted","Data":"f6e193a6c2dd1231ac2255293099be6393a28af58007923e7730f125c978a9e2"} Nov 03 19:39:10.957557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.957498 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podStartSLOduration=1.957479354 podStartE2EDuration="1.957479354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:10.956086174 +0000 UTC m=+717.297199661" watchObservedRunningTime="2026-11-03 19:39:10.957479354 +0000 UTC m=+717.298592841" Nov 03 19:39:10.982472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:10.982417 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podStartSLOduration=1.98239872 podStartE2EDuration="1.98239872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:10.978905611 +0000 UTC m=+717.320019097" watchObservedRunningTime="2026-11-03 19:39:10.98239872 +0000 UTC m=+717.323512209" Nov 03 19:39:11.332890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:11.332856 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:11.334295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:11.334267 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:11.334445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:11.334340 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:11.670548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:11.670451 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:11.672187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:11.672154 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:39:11.672305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:11.672215 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:39:12.335996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:12.335348 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:12.335996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:12.335417 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:12.670837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:12.670755 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:39:12.670837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:12.670804 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:39:13.338372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:13.338332 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:13.338669 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:13.338641 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:13.671207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:13.671122 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:39:13.671207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:13.671170 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:39:14.339396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:14.339355 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:14.339689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:14.339661 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:14.671295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:14.671218 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:39:14.671295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:14.671264 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:39:15.333155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:15.333118 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:15.333579 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:15.333548 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:15.670940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:15.670858 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:39:15.671181 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:15.671157 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:39:16.333327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:16.333289 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:16.333712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:16.333349 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:16.955339 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:16.955307 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:16.956060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:16.956012 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:16.957702 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:16.957679 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:39:17.333486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:17.333448 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:39:17.334217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:17.334178 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:39:18.705991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:18.705200 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:18.706494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:18.706417 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:18.708090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:18.708066 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:39:21.277455 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:39:21.277419 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:39:24.889648 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.889614 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj"] Nov 03 19:39:24.894941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.894920 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.905812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.905786 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj"] Nov 03 19:39:24.912986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.910519 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ef8f84-openshift-default-dockercfg-77kt7\"" Nov 03 19:39:24.949106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949077 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949117 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949151 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949244 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949269 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949290 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949321 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949443 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949405 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzrc\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:24.949515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:24.949497 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051114 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051208 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051236 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051270 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051301 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051338 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051370 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051421 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051449 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzrc\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051693 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051707 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.051989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.051949 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.052082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.052002 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.052316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.052291 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.053553 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.053530 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.053661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.053608 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.065867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.065846 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.066471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.066449 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzrc\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.211890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.211818 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:25.533636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.533587 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj"] Nov 03 19:39:25.552632 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:39:25.551836 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006e3aa8_d443_4fda_9456_9a22856dc123.slice/crio-4571eaa982873dfde2d78c08e2bf4d9c0b95f7080a4c461718611fe4ee805bae WatchSource:0}: Error finding container 4571eaa982873dfde2d78c08e2bf4d9c0b95f7080a4c461718611fe4ee805bae: Status 404 returned error can't find the container with id 4571eaa982873dfde2d78c08e2bf4d9c0b95f7080a4c461718611fe4ee805bae Nov 03 19:39:25.555128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.555090 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:25.555256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.555182 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:25.555256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:25.555228 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:26.089884 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:26.089845 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" event={"ID":"006e3aa8-d443-4fda-9456-9a22856dc123","Type":"ContainerStarted","Data":"08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68"} Nov 03 19:39:26.090341 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:26.089908 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" event={"ID":"006e3aa8-d443-4fda-9456-9a22856dc123","Type":"ContainerStarted","Data":"4571eaa982873dfde2d78c08e2bf4d9c0b95f7080a4c461718611fe4ee805bae"} Nov 03 19:39:26.116708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:26.116663 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podStartSLOduration=2.11665077 podStartE2EDuration="2.11665077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:26.114870049 +0000 UTC m=+732.455983533" watchObservedRunningTime="2026-11-03 19:39:26.11665077 +0000 UTC m=+732.457764276" Nov 03 19:39:26.219998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:26.219348 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:26.222433 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:26.222401 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:39:26.222672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:26.222639 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:39:27.216568 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:27.216525 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:39:27.217124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:27.216595 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:39:28.215395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:28.215358 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:39:28.215742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:28.215710 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:39:29.213446 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:29.213408 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:39:29.214077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:29.214042 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:39:30.214371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:30.214335 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:39:30.215033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:30.215004 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:39:31.238162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:31.238118 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:39:31.238766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:31.238735 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:39:33.250372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:33.250331 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:39:33.250885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:33.250445 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:39:33.326182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:33.326146 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:33.326681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:33.326648 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:33.328012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:33.327960 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:39:49.951451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:49.951411 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l"] Nov 03 19:39:49.957915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:49.957880 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:49.961522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:49.961497 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l"] Nov 03 19:39:49.963185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:49.963165 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:39:49.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:49.963249 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:39:50.070480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.070428 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.070674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.070564 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.070742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.070706 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7sk\" (UniqueName: \"kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.171854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.171806 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7sk\" (UniqueName: \"kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.172023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.172000 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.172205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.172181 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.172820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.172788 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.174734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.174711 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.188599 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.188573 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7sk\" (UniqueName: \"kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk\") pod \"maas-api-revoke-sub-dc1b5af6-j5w7l\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.282177 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.282138 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:50.373521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.372325 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt"] Nov 03 19:39:50.378707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.378679 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.388124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.384711 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt"] Nov 03 19:39:50.477996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.477726 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.477996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.477775 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrhq\" (UniqueName: \"kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.477996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.477822 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.582468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.579803 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.582468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.579856 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrhq\" (UniqueName: \"kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.582468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.579906 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.582468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.580732 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.586486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.586459 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l"] Nov 03 19:39:50.603303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.599207 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrhq\" (UniqueName: \"kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.610007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.607678 5233 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:39:50.610851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.610530 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df422c42-c8vtt\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:50.704715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:50.704683 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:51.050192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.047088 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt"] Nov 03 19:39:51.050192 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:39:51.049580 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc984a88b_530d_4fb8_8b3f_46a11341bbce.slice/crio-45578a096046d525d5eabd5b19e091b28c387c29b518795f8398b0679607f4c9 WatchSource:0}: Error finding container 45578a096046d525d5eabd5b19e091b28c387c29b518795f8398b0679607f4c9: Status 404 returned error can't find the container with id 45578a096046d525d5eabd5b19e091b28c387c29b518795f8398b0679607f4c9 Nov 03 19:39:51.285997 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:39:51.285258 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:39:51.397322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.397288 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr"] Nov 03 19:39:51.405050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.405022 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.412002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.406940 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr"] Nov 03 19:39:51.417785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.417755 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" event={"ID":"da782a86-b292-4ed6-af5a-cb40161cb55a","Type":"ContainerStarted","Data":"30afd9cf73d0ca73f34ca0e2cc48a0e3c70cdf652c0f8be125a5c9597dd00223"} Nov 03 19:39:51.421560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.419765 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" event={"ID":"c984a88b-530d-4fb8-8b3f-46a11341bbce","Type":"ContainerStarted","Data":"45578a096046d525d5eabd5b19e091b28c387c29b518795f8398b0679607f4c9"} Nov 03 19:39:51.501297 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.501261 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.501456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.501377 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.501508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.501476 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhjp\" (UniqueName: \"kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.602889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.602789 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.602889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.602842 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.603164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.602944 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhjp\" (UniqueName: \"kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.603696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.603582 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.605509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.605484 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.617109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.617082 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhjp\" (UniqueName: \"kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp\") pod \"maas-api-revoke-sub-71c31c60-l5tbr\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:51.730994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:51.730081 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:39:52.021731 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.021706 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr"] Nov 03 19:39:52.023800 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:39:52.023773 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf440ca15_e099_4938_a428_34cbda00686d.slice/crio-f4a2aa5e2ea35c42496f8197c4fbfb06e095ae3516efbf1c3e131f239b8d4a4a WatchSource:0}: Error finding container f4a2aa5e2ea35c42496f8197c4fbfb06e095ae3516efbf1c3e131f239b8d4a4a: Status 404 returned error can't find the container with id f4a2aa5e2ea35c42496f8197c4fbfb06e095ae3516efbf1c3e131f239b8d4a4a Nov 03 19:39:52.157984 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.155801 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn"] Nov 03 19:39:52.178709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.176783 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn"] Nov 03 19:39:52.178709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.176909 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.314237 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.313785 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.314237 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.313890 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czg4l\" (UniqueName: \"kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.314237 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.314060 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.417106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.416950 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.417281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.417148 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.417281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.417201 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czg4l\" (UniqueName: \"kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.418403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.418370 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.430361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.426190 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.437044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.431208 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czg4l\" (UniqueName: \"kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l\") pod \"maas-api-revoke-sub-b723e127-5x4jn\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.437044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.434100 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" event={"ID":"f440ca15-e099-4938-a428-34cbda00686d","Type":"ContainerStarted","Data":"f4a2aa5e2ea35c42496f8197c4fbfb06e095ae3516efbf1c3e131f239b8d4a4a"} Nov 03 19:39:52.521995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.521437 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:39:52.861620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:52.860483 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn"] Nov 03 19:39:53.479745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:53.479470 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" event={"ID":"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c","Type":"ContainerStarted","Data":"7d91021e53d06821798ac3ecda6bb86dccc8c128147c09c787bf4f364688591a"} Nov 03 19:39:54.492415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.487615 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9"] Nov 03 19:39:54.504061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.504035 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.510928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.507705 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9"] Nov 03 19:39:54.574719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574686 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.574905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574733 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.574905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574773 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.574905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574815 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2jt\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.574905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574841 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.575165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574907 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.575165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.574965 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.575165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.575021 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.575165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.575049 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.583641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.583597 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-05e217-openshift-default-dockercfg-r9knx\"" Nov 03 19:39:54.676310 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676276 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676340 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676379 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676401 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676513 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676561 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676627 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676693 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2jt\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.676792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.676722 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.677487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.677454 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.679716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.677870 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.679716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.679174 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.679716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.679432 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.679716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.679665 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.680568 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.680524 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.693003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.681457 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.693003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.692910 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2jt\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.697801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.697753 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:54.836577 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:54.835937 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:55.221308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.221268 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:55.221424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.221356 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:55.221424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.221403 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:39:55.222124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.222093 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9"] Nov 03 19:39:55.516928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.516864 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" event={"ID":"c67ddf53-7a1c-43ae-9561-a6956711c692","Type":"ContainerStarted","Data":"b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72"} Nov 03 19:39:55.517595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.516958 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" event={"ID":"c67ddf53-7a1c-43ae-9561-a6956711c692","Type":"ContainerStarted","Data":"fe0a75d49964a92ef627a901ba58ff3cdf6a0bad8d1e1c1c121c3b4d84ef9e4e"} Nov 03 19:39:55.543844 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.543787 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podStartSLOduration=1.543769044 podStartE2EDuration="1.543769044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.540397556 +0000 UTC m=+761.881511067" watchObservedRunningTime="2026-11-03 19:39:55.543769044 +0000 UTC m=+761.884882533" Nov 03 19:39:55.843621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:55.843554 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:39:56.551085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:56.551005 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" event={"ID":"da782a86-b292-4ed6-af5a-cb40161cb55a","Type":"ContainerStarted","Data":"32608cf7e6f022b74f0c47222b32eb4107ec629747df82645a75377fad095e35"} Nov 03 19:39:56.575042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:56.572786 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" podStartSLOduration=2.163064111 podStartE2EDuration="7.572772371s" totalImagesPullingTime="5.40970826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.572255725 +0000 UTC m=+762.913369214" watchObservedRunningTime="2026-11-03 19:39:56.572772371 +0000 UTC m=+762.913885855" Nov 03 19:39:56.839753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:56.839704 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:39:56.839946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:56.839773 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 19:39:57.559059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.559021 5233 generic.go:356] "Generic (PLEG): container finished" podID="c984a88b-530d-4fb8-8b3f-46a11341bbce" containerID="944401aa2afceac4b254893f943168db4dcf648650e6e3e595493f9612cc30ce" exitCode=0 Nov 03 19:39:57.559439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.559121 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" event={"ID":"c984a88b-530d-4fb8-8b3f-46a11341bbce","Type":"ContainerDied","Data":"944401aa2afceac4b254893f943168db4dcf648650e6e3e595493f9612cc30ce"} Nov 03 19:39:57.561357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.561327 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" event={"ID":"f440ca15-e099-4938-a428-34cbda00686d","Type":"ContainerStarted","Data":"ac79bb2b1d7eeefb037e4ace1cf4d0e94b0bc30496baa221dbe14a1b6521db87"} Nov 03 19:39:57.563250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.563229 5233 generic.go:356] "Generic (PLEG): container finished" podID="da782a86-b292-4ed6-af5a-cb40161cb55a" containerID="32608cf7e6f022b74f0c47222b32eb4107ec629747df82645a75377fad095e35" exitCode=0 Nov 03 19:39:57.563609 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.563586 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" event={"ID":"da782a86-b292-4ed6-af5a-cb40161cb55a","Type":"ContainerDied","Data":"32608cf7e6f022b74f0c47222b32eb4107ec629747df82645a75377fad095e35"} Nov 03 19:39:57.632508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.631814 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" podStartSLOduration=1.807312895 podStartE2EDuration="6.631796212s" totalImagesPullingTime="4.824483317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:57.631233501 +0000 UTC m=+763.972346988" watchObservedRunningTime="2026-11-03 19:39:57.631796212 +0000 UTC m=+763.972909698" Nov 03 19:39:57.837192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.837163 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:39:57.837364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:57.837220 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 19:39:58.619489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:58.619401 5233 generic.go:356] "Generic (PLEG): container finished" podID="f440ca15-e099-4938-a428-34cbda00686d" containerID="ac79bb2b1d7eeefb037e4ace1cf4d0e94b0bc30496baa221dbe14a1b6521db87" exitCode=0 Nov 03 19:39:58.620134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:58.620110 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" event={"ID":"f440ca15-e099-4938-a428-34cbda00686d","Type":"ContainerDied","Data":"ac79bb2b1d7eeefb037e4ace1cf4d0e94b0bc30496baa221dbe14a1b6521db87"} Nov 03 19:39:58.671008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:58.667839 5233 generic.go:356] "Generic (PLEG): container finished" podID="3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" containerID="d96e93a97fad56b1389d4fd05b135140369019f140f29a50c405db3d34cd8abc" exitCode=0 Nov 03 19:39:58.671008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:58.668589 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" event={"ID":"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c","Type":"ContainerDied","Data":"d96e93a97fad56b1389d4fd05b135140369019f140f29a50c405db3d34cd8abc"} Nov 03 19:39:58.842993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:58.842303 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:39:58.842993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:58.842379 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 19:39:59.580007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.579934 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:59.677488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.677455 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq\" (UniqueName: \"kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq\") pod \"c984a88b-530d-4fb8-8b3f-46a11341bbce\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " Nov 03 19:39:59.734364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.734324 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq" pod "c984a88b-530d-4fb8-8b3f-46a11341bbce" (UID: "c984a88b-530d-4fb8-8b3f-46a11341bbce"). InnerVolumeSpecName "kube-api-access-2hrhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:39:59.741228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.741198 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" Nov 03 19:39:59.741411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.741372 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt" event={"ID":"c984a88b-530d-4fb8-8b3f-46a11341bbce","Type":"ContainerDied","Data":"45578a096046d525d5eabd5b19e091b28c387c29b518795f8398b0679607f4c9"} Nov 03 19:39:59.741497 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.741435 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45578a096046d525d5eabd5b19e091b28c387c29b518795f8398b0679607f4c9" Nov 03 19:39:59.772998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.771411 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:39:59.778733 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.778692 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca\") pod \"c984a88b-530d-4fb8-8b3f-46a11341bbce\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " Nov 03 19:39:59.779837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.779063 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca\") pod \"c984a88b-530d-4fb8-8b3f-46a11341bbce\" (UID: \"c984a88b-530d-4fb8-8b3f-46a11341bbce\") " Nov 03 19:39:59.779837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.779590 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrhq\" (UniqueName: \"kubernetes.io/projected/c984a88b-530d-4fb8-8b3f-46a11341bbce-kube-api-access-2hrhq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:59.780035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.779965 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca" pod "c984a88b-530d-4fb8-8b3f-46a11341bbce" (UID: "c984a88b-530d-4fb8-8b3f-46a11341bbce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:59.785573 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.782478 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca" pod "c984a88b-530d-4fb8-8b3f-46a11341bbce" (UID: "c984a88b-530d-4fb8-8b3f-46a11341bbce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:39:59.866485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.858444 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:39:59.866485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.858549 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 19:39:59.880254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.880196 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca\") pod \"da782a86-b292-4ed6-af5a-cb40161cb55a\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " Nov 03 19:39:59.880342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.880290 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk\" (UniqueName: \"kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk\") pod \"da782a86-b292-4ed6-af5a-cb40161cb55a\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " Nov 03 19:39:59.880395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.880381 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca\") pod \"da782a86-b292-4ed6-af5a-cb40161cb55a\" (UID: \"da782a86-b292-4ed6-af5a-cb40161cb55a\") " Nov 03 19:39:59.880874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.880856 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:59.880916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.880882 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984a88b-530d-4fb8-8b3f-46a11341bbce-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:59.881329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.881305 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca" pod "da782a86-b292-4ed6-af5a-cb40161cb55a" (UID: "da782a86-b292-4ed6-af5a-cb40161cb55a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:59.884833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.884807 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk" pod "da782a86-b292-4ed6-af5a-cb40161cb55a" (UID: "da782a86-b292-4ed6-af5a-cb40161cb55a"). InnerVolumeSpecName "kube-api-access-hv7sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:39:59.921693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:39:59.921106 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca" pod "da782a86-b292-4ed6-af5a-cb40161cb55a" (UID: "da782a86-b292-4ed6-af5a-cb40161cb55a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:00.002470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.002080 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.002470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.002115 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7sk\" (UniqueName: \"kubernetes.io/projected/da782a86-b292-4ed6-af5a-cb40161cb55a-kube-api-access-hv7sk\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.002470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.002129 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da782a86-b292-4ed6-af5a-cb40161cb55a-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.077674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.077640 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:40:00.138138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.137817 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:40:00.222000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.220349 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca\") pod \"f440ca15-e099-4938-a428-34cbda00686d\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " Nov 03 19:40:00.222000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.220500 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca\") pod \"f440ca15-e099-4938-a428-34cbda00686d\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " Nov 03 19:40:00.222000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.220596 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp\" (UniqueName: \"kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp\") pod \"f440ca15-e099-4938-a428-34cbda00686d\" (UID: \"f440ca15-e099-4938-a428-34cbda00686d\") " Nov 03 19:40:00.222000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.221422 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca" pod "f440ca15-e099-4938-a428-34cbda00686d" (UID: "f440ca15-e099-4938-a428-34cbda00686d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:00.223812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.223761 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp" pod "f440ca15-e099-4938-a428-34cbda00686d" (UID: "f440ca15-e099-4938-a428-34cbda00686d"). InnerVolumeSpecName "kube-api-access-hlhjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:00.223812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.223776 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca" pod "f440ca15-e099-4938-a428-34cbda00686d" (UID: "f440ca15-e099-4938-a428-34cbda00686d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:00.321434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.321400 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca\") pod \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " Nov 03 19:40:00.321608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.321484 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca\") pod \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " Nov 03 19:40:00.321695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.321675 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l\" (UniqueName: \"kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l\") pod \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\" (UID: \"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c\") " Nov 03 19:40:00.321826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.321806 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca" pod "3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" (UID: "3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:00.322191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.322170 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhjp\" (UniqueName: \"kubernetes.io/projected/f440ca15-e099-4938-a428-34cbda00686d-kube-api-access-hlhjp\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.322191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.322190 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f440ca15-e099-4938-a428-34cbda00686d-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.322337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.322199 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.322337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.322208 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f440ca15-e099-4938-a428-34cbda00686d-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.323847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.323820 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca" pod "3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" (UID: "3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:00.324115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.324097 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l" pod "3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" (UID: "3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c"). InnerVolumeSpecName "kube-api-access-czg4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:00.423849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.423755 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czg4l\" (UniqueName: \"kubernetes.io/projected/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-kube-api-access-czg4l\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.423849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.423792 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:00.482000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.481518 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k"] Nov 03 19:40:00.482148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482109 5233 cpu_manager.go:414] "containerMap: removing container" podUID="3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" containerName="revoke-keys" Nov 03 19:40:00.482148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482123 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" containerName="revoke-keys" Nov 03 19:40:00.482148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482147 5233 cpu_manager.go:414] "containerMap: removing container" podUID="da782a86-b292-4ed6-af5a-cb40161cb55a" containerName="revoke-keys" Nov 03 19:40:00.482258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482152 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da782a86-b292-4ed6-af5a-cb40161cb55a" containerName="revoke-keys" Nov 03 19:40:00.482258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482166 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f440ca15-e099-4938-a428-34cbda00686d" containerName="revoke-keys" Nov 03 19:40:00.482258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482172 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f440ca15-e099-4938-a428-34cbda00686d" containerName="revoke-keys" Nov 03 19:40:00.482258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482181 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c984a88b-530d-4fb8-8b3f-46a11341bbce" containerName="revoke-keys" Nov 03 19:40:00.482258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482185 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c984a88b-530d-4fb8-8b3f-46a11341bbce" containerName="revoke-keys" Nov 03 19:40:00.482399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482265 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="da782a86-b292-4ed6-af5a-cb40161cb55a" containerName="revoke-keys" Nov 03 19:40:00.482399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482274 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" containerName="revoke-keys" Nov 03 19:40:00.482399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482281 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f440ca15-e099-4938-a428-34cbda00686d" containerName="revoke-keys" Nov 03 19:40:00.482399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.482287 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c984a88b-530d-4fb8-8b3f-46a11341bbce" containerName="revoke-keys" Nov 03 19:40:00.497207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.497182 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.536471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.534840 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k"] Nov 03 19:40:00.631268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.631223 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvdq\" (UniqueName: \"kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.631809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.631784 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.631935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.631845 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.732781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.732748 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvdq\" (UniqueName: \"kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.733327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.732934 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.733327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.733004 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.733795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.733722 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.735606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.735581 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.746611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.746583 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvdq\" (UniqueName: \"kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq\") pod \"maas-api-revoke-sub-c704263a-qsm8k\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:00.749414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.749392 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" Nov 03 19:40:00.749544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.749418 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l" event={"ID":"da782a86-b292-4ed6-af5a-cb40161cb55a","Type":"ContainerDied","Data":"30afd9cf73d0ca73f34ca0e2cc48a0e3c70cdf652c0f8be125a5c9597dd00223"} Nov 03 19:40:00.749669 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.749568 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30afd9cf73d0ca73f34ca0e2cc48a0e3c70cdf652c0f8be125a5c9597dd00223" Nov 03 19:40:00.750907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.750884 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" Nov 03 19:40:00.750907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.750899 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr" event={"ID":"f440ca15-e099-4938-a428-34cbda00686d","Type":"ContainerDied","Data":"f4a2aa5e2ea35c42496f8197c4fbfb06e095ae3516efbf1c3e131f239b8d4a4a"} Nov 03 19:40:00.751390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.750941 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a2aa5e2ea35c42496f8197c4fbfb06e095ae3516efbf1c3e131f239b8d4a4a" Nov 03 19:40:00.757824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.757803 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" event={"ID":"3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c","Type":"ContainerDied","Data":"7d91021e53d06821798ac3ecda6bb86dccc8c128147c09c787bf4f364688591a"} Nov 03 19:40:00.757917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.757834 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn" Nov 03 19:40:00.757917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.757836 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d91021e53d06821798ac3ecda6bb86dccc8c128147c09c787bf4f364688591a" Nov 03 19:40:00.837376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.837340 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:40:00.837529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.837402 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 19:40:00.859779 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:00.859754 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:01.176684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:01.176657 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k"] Nov 03 19:40:01.178026 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:01.177995 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44e9a18_c94b_4057_bf5e_e67b9cca9af9.slice/crio-1919dc1a9db76f9a4baf41b8b036ed945760c4fb47c085a8acb9fd6a1d38e18d WatchSource:0}: Error finding container 1919dc1a9db76f9a4baf41b8b036ed945760c4fb47c085a8acb9fd6a1d38e18d: Status 404 returned error can't find the container with id 1919dc1a9db76f9a4baf41b8b036ed945760c4fb47c085a8acb9fd6a1d38e18d Nov 03 19:40:01.766274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:01.766237 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" event={"ID":"b44e9a18-c94b-4057-bf5e-e67b9cca9af9","Type":"ContainerStarted","Data":"4a9c08a3f672117ddb479c26fecf0e826a21d89697b2f5991ee5fb3fbeab3743"} Nov 03 19:40:01.766629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:01.766291 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" event={"ID":"b44e9a18-c94b-4057-bf5e-e67b9cca9af9","Type":"ContainerStarted","Data":"1919dc1a9db76f9a4baf41b8b036ed945760c4fb47c085a8acb9fd6a1d38e18d"} Nov 03 19:40:01.787086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:01.787027 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" podStartSLOduration=1.787007426 podStartE2EDuration="1.787007426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:01.784404765 +0000 UTC m=+768.125518251" watchObservedRunningTime="2026-11-03 19:40:01.787007426 +0000 UTC m=+768.128120912" Nov 03 19:40:01.837886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:01.837853 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 19:40:01.838089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:01.837909 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 19:40:02.779316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:02.779282 5233 generic.go:356] "Generic (PLEG): container finished" podID="b44e9a18-c94b-4057-bf5e-e67b9cca9af9" containerID="4a9c08a3f672117ddb479c26fecf0e826a21d89697b2f5991ee5fb3fbeab3743" exitCode=0 Nov 03 19:40:02.779726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:02.779321 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" event={"ID":"b44e9a18-c94b-4057-bf5e-e67b9cca9af9","Type":"ContainerDied","Data":"4a9c08a3f672117ddb479c26fecf0e826a21d89697b2f5991ee5fb3fbeab3743"} Nov 03 19:40:03.164541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:03.164453 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:40:03.165030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:03.164965 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:40:03.169248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:03.169215 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:40:04.274964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.274939 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:04.305764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.305730 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca\") pod \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " Nov 03 19:40:04.305904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.305873 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq\" (UniqueName: \"kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq\") pod \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " Nov 03 19:40:04.305953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.305940 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca\") pod \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\" (UID: \"b44e9a18-c94b-4057-bf5e-e67b9cca9af9\") " Nov 03 19:40:04.306496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.306460 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca" pod "b44e9a18-c94b-4057-bf5e-e67b9cca9af9" (UID: "b44e9a18-c94b-4057-bf5e-e67b9cca9af9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:04.309317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.309289 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca" pod "b44e9a18-c94b-4057-bf5e-e67b9cca9af9" (UID: "b44e9a18-c94b-4057-bf5e-e67b9cca9af9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:04.309561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.309539 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq" pod "b44e9a18-c94b-4057-bf5e-e67b9cca9af9" (UID: "b44e9a18-c94b-4057-bf5e-e67b9cca9af9"). InnerVolumeSpecName "kube-api-access-5qvdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:04.407734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.407658 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qvdq\" (UniqueName: \"kubernetes.io/projected/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-kube-api-access-5qvdq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:04.407734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.407691 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:04.407734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.407702 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44e9a18-c94b-4057-bf5e-e67b9cca9af9-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:04.859994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.859244 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" Nov 03 19:40:04.863139 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.863081 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k" event={"ID":"b44e9a18-c94b-4057-bf5e-e67b9cca9af9","Type":"ContainerDied","Data":"1919dc1a9db76f9a4baf41b8b036ed945760c4fb47c085a8acb9fd6a1d38e18d"} Nov 03 19:40:04.863277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:04.863142 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1919dc1a9db76f9a4baf41b8b036ed945760c4fb47c085a8acb9fd6a1d38e18d" Nov 03 19:40:05.291152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.291118 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp"] Nov 03 19:40:05.291706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.291690 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b44e9a18-c94b-4057-bf5e-e67b9cca9af9" containerName="revoke-keys" Nov 03 19:40:05.291768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.291706 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44e9a18-c94b-4057-bf5e-e67b9cca9af9" containerName="revoke-keys" Nov 03 19:40:05.291838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.291828 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44e9a18-c94b-4057-bf5e-e67b9cca9af9" containerName="revoke-keys" Nov 03 19:40:05.295500 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.295473 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.300850 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.300814 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:40:05.302085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.302062 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:40:05.303872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.303849 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp"] Nov 03 19:40:05.327064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.327033 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdrh\" (UniqueName: \"kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.327209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.327114 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.327209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.327150 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.365132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.365101 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg"] Nov 03 19:40:05.369349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.369327 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.394074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.394036 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg"] Nov 03 19:40:05.428190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428135 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.428190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428198 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.428419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428237 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.428671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428647 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8xg\" (UniqueName: \"kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.428808 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428785 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.428946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428927 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdrh\" (UniqueName: \"kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.429071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.428932 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.431472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.431443 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.453652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.453625 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdrh\" (UniqueName: \"kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh\") pod \"maas-api-revoke-sub-d84a6af3-6h4jp\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.530607 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.530568 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8xg\" (UniqueName: \"kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.530786 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.530678 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.530861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.530838 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.531683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.531653 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.533566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.533543 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.544987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.544914 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8xg\" (UniqueName: \"kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg\") pod \"maas-api-revoke-sub-82262dac-b5lrg\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.622025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.621997 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:05.714539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.714506 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:05.768995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.767771 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg"] Nov 03 19:40:05.774408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.774386 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.785502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.784758 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg"] Nov 03 19:40:05.836549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.836510 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjs4\" (UniqueName: \"kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.836728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.836575 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.836728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.836663 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.927767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.927739 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp"] Nov 03 19:40:05.931273 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:05.931246 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862dd125_578c_4cac_a4a8_cbb2f0d3d4e8.slice/crio-cc696cc89fc5fb8f4e54ca57394301c4959f8a92f16b43fd9babfebef0a5ac19 WatchSource:0}: Error finding container cc696cc89fc5fb8f4e54ca57394301c4959f8a92f16b43fd9babfebef0a5ac19: Status 404 returned error can't find the container with id cc696cc89fc5fb8f4e54ca57394301c4959f8a92f16b43fd9babfebef0a5ac19 Nov 03 19:40:05.937585 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.937565 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjs4\" (UniqueName: \"kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.937651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.937631 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.937802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.937782 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.938378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.938353 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.940940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.940919 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:05.953809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:05.953786 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjs4\" (UniqueName: \"kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4\") pod \"maas-api-revoke-sub-65a1f30d-zjcrg\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:06.021794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.021766 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg"] Nov 03 19:40:06.022966 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:06.022941 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd762f9f_a9dc_4fa1_b1bf_f7973ab5ff15.slice/crio-5da2084268db1b7de8ed6ad89ebb9850aded7dcc049d1f7c4f039262e588aa37 WatchSource:0}: Error finding container 5da2084268db1b7de8ed6ad89ebb9850aded7dcc049d1f7c4f039262e588aa37: Status 404 returned error can't find the container with id 5da2084268db1b7de8ed6ad89ebb9850aded7dcc049d1f7c4f039262e588aa37 Nov 03 19:40:06.099087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.099060 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:06.381149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.381117 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg"] Nov 03 19:40:06.382109 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:06.382082 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8e1af4_cac2_4d73_b0d5_8de6357cba5e.slice/crio-fea931771dd9e9e7a8f1dd99bcaa3d48dccd2209f432140203045dbec8e2d206 WatchSource:0}: Error finding container fea931771dd9e9e7a8f1dd99bcaa3d48dccd2209f432140203045dbec8e2d206: Status 404 returned error can't find the container with id fea931771dd9e9e7a8f1dd99bcaa3d48dccd2209f432140203045dbec8e2d206 Nov 03 19:40:06.875776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.875736 5233 generic.go:356] "Generic (PLEG): container finished" podID="862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" containerID="e51f7d1ba3576d72ba2ed51c29fe188ea916951eb222ffa73bb5ecd4edb58d6c" exitCode=0 Nov 03 19:40:06.875951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.875816 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" event={"ID":"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8","Type":"ContainerDied","Data":"e51f7d1ba3576d72ba2ed51c29fe188ea916951eb222ffa73bb5ecd4edb58d6c"} Nov 03 19:40:06.875951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.875879 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" event={"ID":"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8","Type":"ContainerStarted","Data":"cc696cc89fc5fb8f4e54ca57394301c4959f8a92f16b43fd9babfebef0a5ac19"} Nov 03 19:40:06.879548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.879521 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" event={"ID":"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e","Type":"ContainerStarted","Data":"a28d3281638899de873c5e6ac48d0f07f403ac662190b847667266be32e85b32"} Nov 03 19:40:06.879684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.879572 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" event={"ID":"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e","Type":"ContainerStarted","Data":"fea931771dd9e9e7a8f1dd99bcaa3d48dccd2209f432140203045dbec8e2d206"} Nov 03 19:40:06.881522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.881493 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" event={"ID":"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15","Type":"ContainerStarted","Data":"87d6152acecf97b553469ce0088f9c3eb9c12da2e9bbfecaeb0d4a84d4622ef7"} Nov 03 19:40:06.881677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.881537 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" event={"ID":"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15","Type":"ContainerStarted","Data":"5da2084268db1b7de8ed6ad89ebb9850aded7dcc049d1f7c4f039262e588aa37"} Nov 03 19:40:06.917661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.917611 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" podStartSLOduration=1.917595597 podStartE2EDuration="1.917595597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:06.91534142 +0000 UTC m=+773.256454898" watchObservedRunningTime="2026-11-03 19:40:06.917595597 +0000 UTC m=+773.258709086" Nov 03 19:40:06.940417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:06.938054 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" podStartSLOduration=1.938021613 podStartE2EDuration="1.938021613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:06.933463654 +0000 UTC m=+773.274577140" watchObservedRunningTime="2026-11-03 19:40:06.938021613 +0000 UTC m=+773.279135099" Nov 03 19:40:07.709780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.709698 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx"] Nov 03 19:40:07.713877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.713855 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.720547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.720520 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx"] Nov 03 19:40:07.767280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.767251 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.767417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.767298 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.767417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.767317 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72rk\" (UniqueName: \"kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.868268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.868236 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.868472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.868302 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.868472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.868331 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72rk\" (UniqueName: \"kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.869547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.869520 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.871030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.870965 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.881679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.881657 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72rk\" (UniqueName: \"kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk\") pod \"maas-api-revoke-sub-6236d392-7vdcx\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:07.890760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.890733 5233 generic.go:356] "Generic (PLEG): container finished" podID="dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" containerID="87d6152acecf97b553469ce0088f9c3eb9c12da2e9bbfecaeb0d4a84d4622ef7" exitCode=0 Nov 03 19:40:07.891038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.891012 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" event={"ID":"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15","Type":"ContainerDied","Data":"87d6152acecf97b553469ce0088f9c3eb9c12da2e9bbfecaeb0d4a84d4622ef7"} Nov 03 19:40:07.892802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.892782 5233 generic.go:356] "Generic (PLEG): container finished" podID="8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" containerID="a28d3281638899de873c5e6ac48d0f07f403ac662190b847667266be32e85b32" exitCode=0 Nov 03 19:40:07.892892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:07.892842 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" event={"ID":"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e","Type":"ContainerDied","Data":"a28d3281638899de873c5e6ac48d0f07f403ac662190b847667266be32e85b32"} Nov 03 19:40:08.030203 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.030175 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:08.071301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.071271 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:08.171327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.171286 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh\" (UniqueName: \"kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh\") pod \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " Nov 03 19:40:08.171327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.171321 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca\") pod \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " Nov 03 19:40:08.171567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.171466 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca\") pod \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\" (UID: \"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8\") " Nov 03 19:40:08.171783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.171756 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca" pod "862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" (UID: "862dd125-578c-4cac-a4a8-cbb2f0d3d4e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:08.172152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.172053 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:08.173707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.173677 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh" pod "862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" (UID: "862dd125-578c-4cac-a4a8-cbb2f0d3d4e8"). InnerVolumeSpecName "kube-api-access-pbdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:08.173851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.173826 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca" pod "862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" (UID: "862dd125-578c-4cac-a4a8-cbb2f0d3d4e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:08.273081 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.273051 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbdrh\" (UniqueName: \"kubernetes.io/projected/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-kube-api-access-pbdrh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:08.273081 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.273079 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:08.315094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.315065 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx"] Nov 03 19:40:08.316590 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:08.316564 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3254f83c_ba24_4252_a932_46ee800ccd82.slice/crio-cc871e2765c6e370700b775ef3e6d62f75955ebbcd307a48e4e04ca90b125e68 WatchSource:0}: Error finding container cc871e2765c6e370700b775ef3e6d62f75955ebbcd307a48e4e04ca90b125e68: Status 404 returned error can't find the container with id cc871e2765c6e370700b775ef3e6d62f75955ebbcd307a48e4e04ca90b125e68 Nov 03 19:40:08.902503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.902469 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" event={"ID":"3254f83c-ba24-4252-a932-46ee800ccd82","Type":"ContainerStarted","Data":"ac9972d9d58e40bb3bb40a7361c0861b149412d9bff97b10ddad05b2d51fe152"} Nov 03 19:40:08.902950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.902535 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" event={"ID":"3254f83c-ba24-4252-a932-46ee800ccd82","Type":"ContainerStarted","Data":"cc871e2765c6e370700b775ef3e6d62f75955ebbcd307a48e4e04ca90b125e68"} Nov 03 19:40:08.904117 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.904094 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" Nov 03 19:40:08.904216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.904137 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp" event={"ID":"862dd125-578c-4cac-a4a8-cbb2f0d3d4e8","Type":"ContainerDied","Data":"cc696cc89fc5fb8f4e54ca57394301c4959f8a92f16b43fd9babfebef0a5ac19"} Nov 03 19:40:08.904216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.904181 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc696cc89fc5fb8f4e54ca57394301c4959f8a92f16b43fd9babfebef0a5ac19" Nov 03 19:40:08.922624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:08.922578 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" podStartSLOduration=1.922563834 podStartE2EDuration="1.922563834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:08.920820776 +0000 UTC m=+775.261934262" watchObservedRunningTime="2026-11-03 19:40:08.922563834 +0000 UTC m=+775.263677318" Nov 03 19:40:09.127492 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.127464 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:09.131229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.131171 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:09.183912 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.183881 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4\" (UniqueName: \"kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4\") pod \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " Nov 03 19:40:09.184093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.184026 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca\") pod \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " Nov 03 19:40:09.184168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.184113 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca\") pod \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " Nov 03 19:40:09.184245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.184227 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca\") pod \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " Nov 03 19:40:09.184306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.184276 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg\" (UniqueName: \"kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg\") pod \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\" (UID: \"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15\") " Nov 03 19:40:09.184397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.184377 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca\") pod \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\" (UID: \"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e\") " Nov 03 19:40:09.185027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.184967 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca" pod "8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" (UID: "8b8e1af4-cac2-4d73-b0d5-8de6357cba5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:09.185551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.185516 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca" pod "dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" (UID: "dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:09.191419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.191394 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca" pod "8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" (UID: "8b8e1af4-cac2-4d73-b0d5-8de6357cba5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:09.191537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.191412 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4" pod "8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" (UID: "8b8e1af4-cac2-4d73-b0d5-8de6357cba5e"). InnerVolumeSpecName "kube-api-access-zjjs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:09.191537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.191447 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg" pod "dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" (UID: "dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15"). InnerVolumeSpecName "kube-api-access-th8xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:09.191744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.191725 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca" pod "dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" (UID: "dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:09.286536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.286506 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:09.286536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.286535 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:09.286687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.286545 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:09.286687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.286554 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th8xg\" (UniqueName: \"kubernetes.io/projected/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15-kube-api-access-th8xg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:09.286687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.286562 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:09.286687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.286571 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjs4\" (UniqueName: \"kubernetes.io/projected/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e-kube-api-access-zjjs4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:09.913692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.913652 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" Nov 03 19:40:09.914155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.913688 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg" event={"ID":"8b8e1af4-cac2-4d73-b0d5-8de6357cba5e","Type":"ContainerDied","Data":"fea931771dd9e9e7a8f1dd99bcaa3d48dccd2209f432140203045dbec8e2d206"} Nov 03 19:40:09.914155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.913740 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fea931771dd9e9e7a8f1dd99bcaa3d48dccd2209f432140203045dbec8e2d206" Nov 03 19:40:09.915712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.915681 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" Nov 03 19:40:09.915832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.915753 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg" event={"ID":"dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15","Type":"ContainerDied","Data":"5da2084268db1b7de8ed6ad89ebb9850aded7dcc049d1f7c4f039262e588aa37"} Nov 03 19:40:09.915832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.915797 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da2084268db1b7de8ed6ad89ebb9850aded7dcc049d1f7c4f039262e588aa37" Nov 03 19:40:09.917563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.917539 5233 generic.go:356] "Generic (PLEG): container finished" podID="3254f83c-ba24-4252-a932-46ee800ccd82" containerID="ac9972d9d58e40bb3bb40a7361c0861b149412d9bff97b10ddad05b2d51fe152" exitCode=0 Nov 03 19:40:09.917678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:09.917621 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" event={"ID":"3254f83c-ba24-4252-a932-46ee800ccd82","Type":"ContainerDied","Data":"ac9972d9d58e40bb3bb40a7361c0861b149412d9bff97b10ddad05b2d51fe152"} Nov 03 19:40:11.078780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.078749 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:11.217986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.217941 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca\") pod \"3254f83c-ba24-4252-a932-46ee800ccd82\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " Nov 03 19:40:11.218154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.218088 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca\") pod \"3254f83c-ba24-4252-a932-46ee800ccd82\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " Nov 03 19:40:11.218154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.218142 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk\" (UniqueName: \"kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk\") pod \"3254f83c-ba24-4252-a932-46ee800ccd82\" (UID: \"3254f83c-ba24-4252-a932-46ee800ccd82\") " Nov 03 19:40:11.218348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.218323 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca" pod "3254f83c-ba24-4252-a932-46ee800ccd82" (UID: "3254f83c-ba24-4252-a932-46ee800ccd82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:11.218640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.218622 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.220759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.220730 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca" pod "3254f83c-ba24-4252-a932-46ee800ccd82" (UID: "3254f83c-ba24-4252-a932-46ee800ccd82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:11.220893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.220873 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk" pod "3254f83c-ba24-4252-a932-46ee800ccd82" (UID: "3254f83c-ba24-4252-a932-46ee800ccd82"). InnerVolumeSpecName "kube-api-access-x72rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:11.323434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.323367 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3254f83c-ba24-4252-a932-46ee800ccd82-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.324831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.324800 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72rk\" (UniqueName: \"kubernetes.io/projected/3254f83c-ba24-4252-a932-46ee800ccd82-kube-api-access-x72rk\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.955035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.954955 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" Nov 03 19:40:11.955035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.954954 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx" event={"ID":"3254f83c-ba24-4252-a932-46ee800ccd82","Type":"ContainerDied","Data":"cc871e2765c6e370700b775ef3e6d62f75955ebbcd307a48e4e04ca90b125e68"} Nov 03 19:40:11.955304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:11.955079 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc871e2765c6e370700b775ef3e6d62f75955ebbcd307a48e4e04ca90b125e68" Nov 03 19:40:12.156678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.156598 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9"] Nov 03 19:40:12.157152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157138 5233 cpu_manager.go:414] "containerMap: removing container" podUID="862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" containerName="revoke-keys" Nov 03 19:40:12.157215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157153 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" containerName="revoke-keys" Nov 03 19:40:12.157215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157167 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" containerName="revoke-keys" Nov 03 19:40:12.157215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157172 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" containerName="revoke-keys" Nov 03 19:40:12.157215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157194 5233 cpu_manager.go:414] "containerMap: removing container" podUID="3254f83c-ba24-4252-a932-46ee800ccd82" containerName="revoke-keys" Nov 03 19:40:12.157215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157199 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3254f83c-ba24-4252-a932-46ee800ccd82" containerName="revoke-keys" Nov 03 19:40:12.157356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157221 5233 cpu_manager.go:414] "containerMap: removing container" podUID="dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" containerName="revoke-keys" Nov 03 19:40:12.157356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157226 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" containerName="revoke-keys" Nov 03 19:40:12.157356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157300 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" containerName="revoke-keys" Nov 03 19:40:12.157356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157309 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" containerName="revoke-keys" Nov 03 19:40:12.157356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157317 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="3254f83c-ba24-4252-a932-46ee800ccd82" containerName="revoke-keys" Nov 03 19:40:12.157356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.157325 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" containerName="revoke-keys" Nov 03 19:40:12.160964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.160948 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.165703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.165679 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:40:12.165941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.165919 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:40:12.168838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.168812 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9"] Nov 03 19:40:12.338112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.338075 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.338294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.338146 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.338294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.338273 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsn2k\" (UniqueName: \"kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.439819 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.439742 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.439819 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.439794 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.440012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.439827 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsn2k\" (UniqueName: \"kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.440382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.440362 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.442268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.442245 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.452955 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.452934 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsn2k\" (UniqueName: \"kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k\") pod \"maas-api-revoke-sub-6e8e24c3-nnlq9\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.476918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.476894 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:12.770629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.770599 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9"] Nov 03 19:40:12.770929 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:12.770902 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d45166_f204_4f00_95c6_f97de55c5ac4.slice/crio-37d870dab2d86fb7a6ee0da82230c46b42781004141fc2a7ceab5a5e3769c030 WatchSource:0}: Error finding container 37d870dab2d86fb7a6ee0da82230c46b42781004141fc2a7ceab5a5e3769c030: Status 404 returned error can't find the container with id 37d870dab2d86fb7a6ee0da82230c46b42781004141fc2a7ceab5a5e3769c030 Nov 03 19:40:12.979022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:12.974190 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" event={"ID":"a1d45166-f204-4f00-95c6-f97de55c5ac4","Type":"ContainerStarted","Data":"37d870dab2d86fb7a6ee0da82230c46b42781004141fc2a7ceab5a5e3769c030"} Nov 03 19:40:14.176997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:14.176352 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" event={"ID":"a1d45166-f204-4f00-95c6-f97de55c5ac4","Type":"ContainerStarted","Data":"6293a9fef9b066cc3df992db219c2f27e2fecfee9473c1122a686620724234e4"} Nov 03 19:40:14.236995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:14.236509 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" podStartSLOduration=2.236489276 podStartE2EDuration="2.236489276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:14.235506932 +0000 UTC m=+780.576620457" watchObservedRunningTime="2026-11-03 19:40:14.236489276 +0000 UTC m=+780.577602763" Nov 03 19:40:14.554994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:14.552055 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:14.672996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:14.672061 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:14.935180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:14.930144 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:15.236995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.234635 5233 generic.go:356] "Generic (PLEG): container finished" podID="a1d45166-f204-4f00-95c6-f97de55c5ac4" containerID="6293a9fef9b066cc3df992db219c2f27e2fecfee9473c1122a686620724234e4" exitCode=0 Nov 03 19:40:15.236995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.234687 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" event={"ID":"a1d45166-f204-4f00-95c6-f97de55c5ac4","Type":"ContainerDied","Data":"6293a9fef9b066cc3df992db219c2f27e2fecfee9473c1122a686620724234e4"} Nov 03 19:40:15.610108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.610023 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8"] Nov 03 19:40:15.688042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.685253 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.733017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.695820 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8"] Nov 03 19:40:15.808351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.808318 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.808649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.808625 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5r24\" (UniqueName: \"kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.809140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.809109 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.958099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.924699 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.958099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.924789 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5r24\" (UniqueName: \"kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.958099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.924990 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:15.958099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:15.950893 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:16.062094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.007415 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:16.062094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.025937 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5r24\" (UniqueName: \"kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24\") pod \"maas-api-revoke-sub-d4534906-45rw8\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:16.062094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.054022 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx"] Nov 03 19:40:16.155632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.154917 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.158518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.158492 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx"] Nov 03 19:40:16.318197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.318163 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:16.502430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.459754 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.502430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.459828 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2sz\" (UniqueName: \"kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.502430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.460029 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.550702 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.549051 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:16.596078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.588729 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.596078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.588793 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2sz\" (UniqueName: \"kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.596078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.588950 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.596078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.592808 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.649141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.648750 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2sz\" (UniqueName: \"kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.669595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.669518 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b67f5d9-sjhxx\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:16.696127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.695526 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:16.758295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:16.758262 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:17.112048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.111749 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:17.546619 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.546585 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8"] Nov 03 19:40:17.598039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.595738 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:17.650060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.650029 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca\") pod \"a1d45166-f204-4f00-95c6-f97de55c5ac4\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " Nov 03 19:40:17.650222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.650189 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k\" (UniqueName: \"kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k\") pod \"a1d45166-f204-4f00-95c6-f97de55c5ac4\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " Nov 03 19:40:17.650476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.650431 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca" pod "a1d45166-f204-4f00-95c6-f97de55c5ac4" (UID: "a1d45166-f204-4f00-95c6-f97de55c5ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:17.652733 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.652705 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k" pod "a1d45166-f204-4f00-95c6-f97de55c5ac4" (UID: "a1d45166-f204-4f00-95c6-f97de55c5ac4"). InnerVolumeSpecName "kube-api-access-zsn2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:17.678153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.678091 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx"] Nov 03 19:40:17.679584 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:17.679561 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69293c2_5614_4aec_9d76_c589f1ff7c81.slice/crio-8fe0ea0417c6f27970666d3aea0cb21409f864817576f752d5b31864a941d04d WatchSource:0}: Error finding container 8fe0ea0417c6f27970666d3aea0cb21409f864817576f752d5b31864a941d04d: Status 404 returned error can't find the container with id 8fe0ea0417c6f27970666d3aea0cb21409f864817576f752d5b31864a941d04d Nov 03 19:40:17.751535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.751504 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca\") pod \"a1d45166-f204-4f00-95c6-f97de55c5ac4\" (UID: \"a1d45166-f204-4f00-95c6-f97de55c5ac4\") " Nov 03 19:40:17.752253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.752226 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsn2k\" (UniqueName: \"kubernetes.io/projected/a1d45166-f204-4f00-95c6-f97de55c5ac4-kube-api-access-zsn2k\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:17.752303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.752263 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:17.753405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.753380 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca" pod "a1d45166-f204-4f00-95c6-f97de55c5ac4" (UID: "a1d45166-f204-4f00-95c6-f97de55c5ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:17.853475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:17.853387 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d45166-f204-4f00-95c6-f97de55c5ac4-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:18.532300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.532273 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" Nov 03 19:40:18.532456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.532269 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9" event={"ID":"a1d45166-f204-4f00-95c6-f97de55c5ac4","Type":"ContainerDied","Data":"37d870dab2d86fb7a6ee0da82230c46b42781004141fc2a7ceab5a5e3769c030"} Nov 03 19:40:18.532456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.532413 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d870dab2d86fb7a6ee0da82230c46b42781004141fc2a7ceab5a5e3769c030" Nov 03 19:40:18.534126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.534098 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" event={"ID":"7a59d344-e0cb-4b55-95df-94d4e140192e","Type":"ContainerStarted","Data":"c565dbc76c178bc2cec4b6a008a4e7d16220e9bdf5653af32548841db592b1a1"} Nov 03 19:40:18.534243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.534143 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" event={"ID":"7a59d344-e0cb-4b55-95df-94d4e140192e","Type":"ContainerStarted","Data":"b5347490199885f31c8fceabcf072d66b520752113ac750c185048d860e75e77"} Nov 03 19:40:18.535957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.535933 5233 generic.go:356] "Generic (PLEG): container finished" podID="e69293c2-5614-4aec-9d76-c589f1ff7c81" containerID="e655b3de9a2103ad225b74ed38a588474153e899304e0bf33422b71af3092d99" exitCode=0 Nov 03 19:40:18.536076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.536027 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" event={"ID":"e69293c2-5614-4aec-9d76-c589f1ff7c81","Type":"ContainerDied","Data":"e655b3de9a2103ad225b74ed38a588474153e899304e0bf33422b71af3092d99"} Nov 03 19:40:18.536076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.536067 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" event={"ID":"e69293c2-5614-4aec-9d76-c589f1ff7c81","Type":"ContainerStarted","Data":"8fe0ea0417c6f27970666d3aea0cb21409f864817576f752d5b31864a941d04d"} Nov 03 19:40:18.556845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.556822 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt"] Nov 03 19:40:18.557566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.557547 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a1d45166-f204-4f00-95c6-f97de55c5ac4" containerName="revoke-keys" Nov 03 19:40:18.557662 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.557569 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d45166-f204-4f00-95c6-f97de55c5ac4" containerName="revoke-keys" Nov 03 19:40:18.557761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.557747 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d45166-f204-4f00-95c6-f97de55c5ac4" containerName="revoke-keys" Nov 03 19:40:18.561309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.561267 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" podStartSLOduration=3.561255399 podStartE2EDuration="3.561255399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:18.559774088 +0000 UTC m=+784.900887575" watchObservedRunningTime="2026-11-03 19:40:18.561255399 +0000 UTC m=+784.902368883" Nov 03 19:40:18.561404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.561317 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.570771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.570718 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt"] Nov 03 19:40:18.661938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.661910 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpr46\" (UniqueName: \"kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.662078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.662025 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.662202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.662186 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.763599 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.763527 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.763745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.763666 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.763745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.763707 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpr46\" (UniqueName: \"kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.764353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.764324 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.765849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.765827 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.782928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.782903 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpr46\" (UniqueName: \"kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46\") pod \"maas-api-revoke-sub-20f7c26f-5ghdt\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:18.878428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:18.878388 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:19.161859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.161828 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt"] Nov 03 19:40:19.162467 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:19.162437 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b47b081_a69a_482b_9422_4201f9db2343.slice/crio-71507dbe890f2da2bfeb98208d1a0145dc470040befb09f3c1cba3559e4ee4f2 WatchSource:0}: Error finding container 71507dbe890f2da2bfeb98208d1a0145dc470040befb09f3c1cba3559e4ee4f2: Status 404 returned error can't find the container with id 71507dbe890f2da2bfeb98208d1a0145dc470040befb09f3c1cba3559e4ee4f2 Nov 03 19:40:19.545549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.545510 5233 generic.go:356] "Generic (PLEG): container finished" podID="7a59d344-e0cb-4b55-95df-94d4e140192e" containerID="c565dbc76c178bc2cec4b6a008a4e7d16220e9bdf5653af32548841db592b1a1" exitCode=0 Nov 03 19:40:19.545719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.545589 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" event={"ID":"7a59d344-e0cb-4b55-95df-94d4e140192e","Type":"ContainerDied","Data":"c565dbc76c178bc2cec4b6a008a4e7d16220e9bdf5653af32548841db592b1a1"} Nov 03 19:40:19.547465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.547439 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" event={"ID":"4b47b081-a69a-482b-9422-4201f9db2343","Type":"ContainerStarted","Data":"0626c91e837bbc6626b854b074291382c3d3c5c952cbc75be846604eda4aba6e"} Nov 03 19:40:19.547570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.547485 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" event={"ID":"4b47b081-a69a-482b-9422-4201f9db2343","Type":"ContainerStarted","Data":"71507dbe890f2da2bfeb98208d1a0145dc470040befb09f3c1cba3559e4ee4f2"} Nov 03 19:40:19.585547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.585487 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" podStartSLOduration=1.585468427 podStartE2EDuration="1.585468427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:19.584015318 +0000 UTC m=+785.925128805" watchObservedRunningTime="2026-11-03 19:40:19.585468427 +0000 UTC m=+785.926581913" Nov 03 19:40:19.712919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.712894 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:19.774959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.774872 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca\") pod \"e69293c2-5614-4aec-9d76-c589f1ff7c81\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " Nov 03 19:40:19.774959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.774953 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca\") pod \"e69293c2-5614-4aec-9d76-c589f1ff7c81\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " Nov 03 19:40:19.775394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.775024 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz\" (UniqueName: \"kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz\") pod \"e69293c2-5614-4aec-9d76-c589f1ff7c81\" (UID: \"e69293c2-5614-4aec-9d76-c589f1ff7c81\") " Nov 03 19:40:19.776419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.776389 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca" pod "e69293c2-5614-4aec-9d76-c589f1ff7c81" (UID: "e69293c2-5614-4aec-9d76-c589f1ff7c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:19.778472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.778450 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca" pod "e69293c2-5614-4aec-9d76-c589f1ff7c81" (UID: "e69293c2-5614-4aec-9d76-c589f1ff7c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:19.778810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.778788 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz" pod "e69293c2-5614-4aec-9d76-c589f1ff7c81" (UID: "e69293c2-5614-4aec-9d76-c589f1ff7c81"). InnerVolumeSpecName "kube-api-access-bj2sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:19.876941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.876903 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:19.876941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.876931 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2sz\" (UniqueName: \"kubernetes.io/projected/e69293c2-5614-4aec-9d76-c589f1ff7c81-kube-api-access-bj2sz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:19.877152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:19.876949 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69293c2-5614-4aec-9d76-c589f1ff7c81-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:20.011027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.010992 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj"] Nov 03 19:40:20.011985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.011940 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e69293c2-5614-4aec-9d76-c589f1ff7c81" containerName="revoke-keys" Nov 03 19:40:20.011985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.011985 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69293c2-5614-4aec-9d76-c589f1ff7c81" containerName="revoke-keys" Nov 03 19:40:20.012180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.012164 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69293c2-5614-4aec-9d76-c589f1ff7c81" containerName="revoke-keys" Nov 03 19:40:20.023032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.022712 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj"] Nov 03 19:40:20.023032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.022914 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.079726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.079692 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.079905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.079882 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j2j\" (UniqueName: \"kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.079944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.079933 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.180754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.180724 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.180906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.180861 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j2j\" (UniqueName: \"kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.180906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.180901 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.181567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.181544 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.183082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.183053 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.204695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.204666 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j2j\" (UniqueName: \"kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j\") pod \"maas-api-revoke-sub-16f2395a-fmhdj\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.341909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.341827 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:20.347680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.347654 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p"] Nov 03 19:40:20.352896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.352877 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.366694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.366571 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p"] Nov 03 19:40:20.387140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.387111 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7674j\" (UniqueName: \"kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.387266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.387166 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.387266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.387232 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.488424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.488387 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7674j\" (UniqueName: \"kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.488582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.488446 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.488582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.488508 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.489232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.489207 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.490770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.490746 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.500830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.500807 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7674j\" (UniqueName: \"kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j\") pod \"maas-api-revoke-sub-cec52661-22k5p\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.559717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.559681 5233 generic.go:356] "Generic (PLEG): container finished" podID="4b47b081-a69a-482b-9422-4201f9db2343" containerID="0626c91e837bbc6626b854b074291382c3d3c5c952cbc75be846604eda4aba6e" exitCode=0 Nov 03 19:40:20.559897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.559778 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" event={"ID":"4b47b081-a69a-482b-9422-4201f9db2343","Type":"ContainerDied","Data":"0626c91e837bbc6626b854b074291382c3d3c5c952cbc75be846604eda4aba6e"} Nov 03 19:40:20.562225 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.562196 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" Nov 03 19:40:20.562915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.562885 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx" event={"ID":"e69293c2-5614-4aec-9d76-c589f1ff7c81","Type":"ContainerDied","Data":"8fe0ea0417c6f27970666d3aea0cb21409f864817576f752d5b31864a941d04d"} Nov 03 19:40:20.563058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.562935 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fe0ea0417c6f27970666d3aea0cb21409f864817576f752d5b31864a941d04d" Nov 03 19:40:20.650437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.650405 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj"] Nov 03 19:40:20.652445 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:20.652416 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05393ae_72bf_48b4_b871_c8c59d6e4301.slice/crio-ddf4e3892a7851dc186d0c9f43dd947ad9636b73d0204bd049a90f330b4590f1 WatchSource:0}: Error finding container ddf4e3892a7851dc186d0c9f43dd947ad9636b73d0204bd049a90f330b4590f1: Status 404 returned error can't find the container with id ddf4e3892a7851dc186d0c9f43dd947ad9636b73d0204bd049a90f330b4590f1 Nov 03 19:40:20.687953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.687923 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:20.796927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.796900 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:20.893191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.893110 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca\") pod \"7a59d344-e0cb-4b55-95df-94d4e140192e\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " Nov 03 19:40:20.893489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.893338 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca\") pod \"7a59d344-e0cb-4b55-95df-94d4e140192e\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " Nov 03 19:40:20.893489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.893406 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24\" (UniqueName: \"kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24\") pod \"7a59d344-e0cb-4b55-95df-94d4e140192e\" (UID: \"7a59d344-e0cb-4b55-95df-94d4e140192e\") " Nov 03 19:40:20.894421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.893735 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca" pod "7a59d344-e0cb-4b55-95df-94d4e140192e" (UID: "7a59d344-e0cb-4b55-95df-94d4e140192e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:20.894421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.893963 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:20.895637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.895612 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca" pod "7a59d344-e0cb-4b55-95df-94d4e140192e" (UID: "7a59d344-e0cb-4b55-95df-94d4e140192e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:20.896271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.895995 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24" pod "7a59d344-e0cb-4b55-95df-94d4e140192e" (UID: "7a59d344-e0cb-4b55-95df-94d4e140192e"). InnerVolumeSpecName "kube-api-access-d5r24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:20.995001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.994960 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5r24\" (UniqueName: \"kubernetes.io/projected/7a59d344-e0cb-4b55-95df-94d4e140192e-kube-api-access-d5r24\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:20.995145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:20.995005 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a59d344-e0cb-4b55-95df-94d4e140192e-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:21.062148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.062117 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p"] Nov 03 19:40:21.063282 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:21.063251 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48a00f8_5c3d_463b_b3fb_a7ff07fcca04.slice/crio-e48d352ee696ca485b934cecd038e5259e96b857c8ea44b45e28031c259e526a WatchSource:0}: Error finding container e48d352ee696ca485b934cecd038e5259e96b857c8ea44b45e28031c259e526a: Status 404 returned error can't find the container with id e48d352ee696ca485b934cecd038e5259e96b857c8ea44b45e28031c259e526a Nov 03 19:40:21.277299 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:40:21.277252 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:40:21.571886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.571805 5233 generic.go:356] "Generic (PLEG): container finished" podID="a05393ae-72bf-48b4-b871-c8c59d6e4301" containerID="2af3feff03749aeb1e5dc62389634a21287a2f680a94ce7458ad5bf7c554a850" exitCode=0 Nov 03 19:40:21.571886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.571848 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" event={"ID":"a05393ae-72bf-48b4-b871-c8c59d6e4301","Type":"ContainerDied","Data":"2af3feff03749aeb1e5dc62389634a21287a2f680a94ce7458ad5bf7c554a850"} Nov 03 19:40:21.572108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.571911 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" event={"ID":"a05393ae-72bf-48b4-b871-c8c59d6e4301","Type":"ContainerStarted","Data":"ddf4e3892a7851dc186d0c9f43dd947ad9636b73d0204bd049a90f330b4590f1"} Nov 03 19:40:21.573479 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.573459 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" Nov 03 19:40:21.573608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.573495 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8" event={"ID":"7a59d344-e0cb-4b55-95df-94d4e140192e","Type":"ContainerDied","Data":"b5347490199885f31c8fceabcf072d66b520752113ac750c185048d860e75e77"} Nov 03 19:40:21.573608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.573538 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5347490199885f31c8fceabcf072d66b520752113ac750c185048d860e75e77" Nov 03 19:40:21.575287 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.575261 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" event={"ID":"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04","Type":"ContainerStarted","Data":"fe8613d97eb0759428be03ed68c0e7972bbc276308a66fa59ef63e6140fad66a"} Nov 03 19:40:21.575391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.575312 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" event={"ID":"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04","Type":"ContainerStarted","Data":"e48d352ee696ca485b934cecd038e5259e96b857c8ea44b45e28031c259e526a"} Nov 03 19:40:21.736835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.736810 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:21.757157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.755765 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" podStartSLOduration=1.755751578 podStartE2EDuration="1.755751578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:21.618210977 +0000 UTC m=+787.959324463" watchObservedRunningTime="2026-11-03 19:40:21.755751578 +0000 UTC m=+788.096865063" Nov 03 19:40:21.804494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.804453 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca\") pod \"4b47b081-a69a-482b-9422-4201f9db2343\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " Nov 03 19:40:21.804494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.804499 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46\" (UniqueName: \"kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46\") pod \"4b47b081-a69a-482b-9422-4201f9db2343\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " Nov 03 19:40:21.804737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.804603 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca\") pod \"4b47b081-a69a-482b-9422-4201f9db2343\" (UID: \"4b47b081-a69a-482b-9422-4201f9db2343\") " Nov 03 19:40:21.804846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.804823 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca" pod "4b47b081-a69a-482b-9422-4201f9db2343" (UID: "4b47b081-a69a-482b-9422-4201f9db2343"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:21.805196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.805174 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:21.806659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.806634 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca" pod "4b47b081-a69a-482b-9422-4201f9db2343" (UID: "4b47b081-a69a-482b-9422-4201f9db2343"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:21.806758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.806656 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46" pod "4b47b081-a69a-482b-9422-4201f9db2343" (UID: "4b47b081-a69a-482b-9422-4201f9db2343"). InnerVolumeSpecName "kube-api-access-xpr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:21.906193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.906161 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpr46\" (UniqueName: \"kubernetes.io/projected/4b47b081-a69a-482b-9422-4201f9db2343-kube-api-access-xpr46\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:21.906361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:21.906339 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b47b081-a69a-482b-9422-4201f9db2343-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:22.584660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.584620 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" event={"ID":"4b47b081-a69a-482b-9422-4201f9db2343","Type":"ContainerDied","Data":"71507dbe890f2da2bfeb98208d1a0145dc470040befb09f3c1cba3559e4ee4f2"} Nov 03 19:40:22.584660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.584654 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt" Nov 03 19:40:22.585023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.584683 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71507dbe890f2da2bfeb98208d1a0145dc470040befb09f3c1cba3559e4ee4f2" Nov 03 19:40:22.586415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.586389 5233 generic.go:356] "Generic (PLEG): container finished" podID="e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" containerID="fe8613d97eb0759428be03ed68c0e7972bbc276308a66fa59ef63e6140fad66a" exitCode=0 Nov 03 19:40:22.586523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.586447 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" event={"ID":"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04","Type":"ContainerDied","Data":"fe8613d97eb0759428be03ed68c0e7972bbc276308a66fa59ef63e6140fad66a"} Nov 03 19:40:22.783767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.783733 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:22.815817 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.815774 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j\" (UniqueName: \"kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j\") pod \"a05393ae-72bf-48b4-b871-c8c59d6e4301\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " Nov 03 19:40:22.816018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.815854 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca\") pod \"a05393ae-72bf-48b4-b871-c8c59d6e4301\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " Nov 03 19:40:22.816097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.816045 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca\") pod \"a05393ae-72bf-48b4-b871-c8c59d6e4301\" (UID: \"a05393ae-72bf-48b4-b871-c8c59d6e4301\") " Nov 03 19:40:22.816294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.816265 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca" pod "a05393ae-72bf-48b4-b871-c8c59d6e4301" (UID: "a05393ae-72bf-48b4-b871-c8c59d6e4301"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:22.816654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.816630 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:22.818083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.818057 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j" pod "a05393ae-72bf-48b4-b871-c8c59d6e4301" (UID: "a05393ae-72bf-48b4-b871-c8c59d6e4301"). InnerVolumeSpecName "kube-api-access-r6j2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:22.818543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.818520 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca" pod "a05393ae-72bf-48b4-b871-c8c59d6e4301" (UID: "a05393ae-72bf-48b4-b871-c8c59d6e4301"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:22.917886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.917792 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6j2j\" (UniqueName: \"kubernetes.io/projected/a05393ae-72bf-48b4-b871-c8c59d6e4301-kube-api-access-r6j2j\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:22.917886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:22.917822 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05393ae-72bf-48b4-b871-c8c59d6e4301-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:23.597062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.597028 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" Nov 03 19:40:23.597062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.597057 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj" event={"ID":"a05393ae-72bf-48b4-b871-c8c59d6e4301","Type":"ContainerDied","Data":"ddf4e3892a7851dc186d0c9f43dd947ad9636b73d0204bd049a90f330b4590f1"} Nov 03 19:40:23.597334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.597104 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf4e3892a7851dc186d0c9f43dd947ad9636b73d0204bd049a90f330b4590f1" Nov 03 19:40:23.759694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.759671 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:23.828121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.828090 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca\") pod \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " Nov 03 19:40:23.828604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.828311 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j\" (UniqueName: \"kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j\") pod \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " Nov 03 19:40:23.828604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.828352 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca\") pod \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\" (UID: \"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04\") " Nov 03 19:40:23.831217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.829230 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca" pod "e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" (UID: "e48a00f8-5c3d-463b-b3fb-a7ff07fcca04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:23.831360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.831334 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca" pod "e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" (UID: "e48a00f8-5c3d-463b-b3fb-a7ff07fcca04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:23.833206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.833172 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j" pod "e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" (UID: "e48a00f8-5c3d-463b-b3fb-a7ff07fcca04"). InnerVolumeSpecName "kube-api-access-7674j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:23.930194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.930161 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7674j\" (UniqueName: \"kubernetes.io/projected/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-kube-api-access-7674j\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:23.930194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.930192 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:23.930371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:23.930206 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:24.606088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:24.606046 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" event={"ID":"e48a00f8-5c3d-463b-b3fb-a7ff07fcca04","Type":"ContainerDied","Data":"e48d352ee696ca485b934cecd038e5259e96b857c8ea44b45e28031c259e526a"} Nov 03 19:40:24.606088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:24.606079 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p" Nov 03 19:40:24.606348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:24.606142 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48d352ee696ca485b934cecd038e5259e96b857c8ea44b45e28031c259e526a" Nov 03 19:40:25.895405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.895368 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8"] Nov 03 19:40:25.896283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896264 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a05393ae-72bf-48b4-b871-c8c59d6e4301" containerName="revoke-keys" Nov 03 19:40:25.896326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896286 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05393ae-72bf-48b4-b871-c8c59d6e4301" containerName="revoke-keys" Nov 03 19:40:25.896326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896306 5233 cpu_manager.go:414] "containerMap: removing container" podUID="7a59d344-e0cb-4b55-95df-94d4e140192e" containerName="revoke-keys" Nov 03 19:40:25.896326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896313 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a59d344-e0cb-4b55-95df-94d4e140192e" containerName="revoke-keys" Nov 03 19:40:25.896430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896326 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" containerName="revoke-keys" Nov 03 19:40:25.896430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896335 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" containerName="revoke-keys" Nov 03 19:40:25.896430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896393 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4b47b081-a69a-482b-9422-4201f9db2343" containerName="revoke-keys" Nov 03 19:40:25.896430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896400 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b47b081-a69a-482b-9422-4201f9db2343" containerName="revoke-keys" Nov 03 19:40:25.896545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896531 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b47b081-a69a-482b-9422-4201f9db2343" containerName="revoke-keys" Nov 03 19:40:25.896574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896545 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a59d344-e0cb-4b55-95df-94d4e140192e" containerName="revoke-keys" Nov 03 19:40:25.896574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896554 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05393ae-72bf-48b4-b871-c8c59d6e4301" containerName="revoke-keys" Nov 03 19:40:25.896574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.896564 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" containerName="revoke-keys" Nov 03 19:40:25.906944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.906903 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8"] Nov 03 19:40:25.907157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.907038 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:25.917874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.917838 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:40:25.920060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.920040 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:40:25.962507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.962482 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:25.962652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.962534 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jdn\" (UniqueName: \"kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:25.962728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:25.962699 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.043171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.043139 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72"] Nov 03 19:40:26.047338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.047313 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.055649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.055622 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72"] Nov 03 19:40:26.064434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.064411 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.064552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.064506 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.064552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.064541 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jdn\" (UniqueName: \"kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.065706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.065674 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.067028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.067001 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.077155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.077129 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jdn\" (UniqueName: \"kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn\") pod \"maas-api-revoke-sub-3be2bdbe-vzlg8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.165448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.165348 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.165638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.165531 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.165734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.165717 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcxm\" (UniqueName: \"kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.237654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.237619 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:26.267049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.267005 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcxm\" (UniqueName: \"kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.267186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.267098 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.267262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.267187 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.267832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.267807 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.269253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.269232 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.283840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.283818 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcxm\" (UniqueName: \"kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm\") pod \"maas-api-revoke-sub-3da62dd3-cfm72\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.369242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.368082 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:26.524856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.524799 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8"] Nov 03 19:40:26.539781 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:26.539737 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5dcbd5_ee40_4876_a3c6_4a699dd848e8.slice/crio-8012d09a710ccfa438f45583d611075f5121c21ee27001f46d289183b24bc65c WatchSource:0}: Error finding container 8012d09a710ccfa438f45583d611075f5121c21ee27001f46d289183b24bc65c: Status 404 returned error can't find the container with id 8012d09a710ccfa438f45583d611075f5121c21ee27001f46d289183b24bc65c Nov 03 19:40:26.627581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.627545 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" event={"ID":"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8","Type":"ContainerStarted","Data":"8012d09a710ccfa438f45583d611075f5121c21ee27001f46d289183b24bc65c"} Nov 03 19:40:26.668574 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:26.668543 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf726797a_52fd_44d8_b5a4_d9cd42446fd8.slice/crio-a87f4f49e189d8e40f6fdbb83da85970eea82c3a803bca48dfdf518660181389 WatchSource:0}: Error finding container a87f4f49e189d8e40f6fdbb83da85970eea82c3a803bca48dfdf518660181389: Status 404 returned error can't find the container with id a87f4f49e189d8e40f6fdbb83da85970eea82c3a803bca48dfdf518660181389 Nov 03 19:40:26.668574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:26.668570 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72"] Nov 03 19:40:27.638062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:27.638031 5233 generic.go:356] "Generic (PLEG): container finished" podID="2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" containerID="8df39b38f1cd18dfc88bda8aec9d1ba7b6c992df22ecc9bd11f71c630c339f2c" exitCode=0 Nov 03 19:40:27.638602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:27.638116 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" event={"ID":"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8","Type":"ContainerDied","Data":"8df39b38f1cd18dfc88bda8aec9d1ba7b6c992df22ecc9bd11f71c630c339f2c"} Nov 03 19:40:27.644719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:27.644686 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" event={"ID":"f726797a-52fd-44d8-b5a4-d9cd42446fd8","Type":"ContainerStarted","Data":"9b36d20ec7d03c741518688ed2c5d6914cae0420b8802c4912208dbdbc9e30a1"} Nov 03 19:40:27.644847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:27.644739 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" event={"ID":"f726797a-52fd-44d8-b5a4-d9cd42446fd8","Type":"ContainerStarted","Data":"a87f4f49e189d8e40f6fdbb83da85970eea82c3a803bca48dfdf518660181389"} Nov 03 19:40:27.681947 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:27.681902 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" podStartSLOduration=1.681889098 podStartE2EDuration="1.681889098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:27.679758779 +0000 UTC m=+794.020872265" watchObservedRunningTime="2026-11-03 19:40:27.681889098 +0000 UTC m=+794.023002582" Nov 03 19:40:28.696996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:28.695575 5233 generic.go:356] "Generic (PLEG): container finished" podID="f726797a-52fd-44d8-b5a4-d9cd42446fd8" containerID="9b36d20ec7d03c741518688ed2c5d6914cae0420b8802c4912208dbdbc9e30a1" exitCode=0 Nov 03 19:40:28.696996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:28.696309 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" event={"ID":"f726797a-52fd-44d8-b5a4-d9cd42446fd8","Type":"ContainerDied","Data":"9b36d20ec7d03c741518688ed2c5d6914cae0420b8802c4912208dbdbc9e30a1"} Nov 03 19:40:29.871335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:29.870736 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:29.959659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:29.959601 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:29.998205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:29.998173 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:30.064076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.059447 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" Nov 03 19:40:30.088014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.062112 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8" event={"ID":"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8","Type":"ContainerDied","Data":"8012d09a710ccfa438f45583d611075f5121c21ee27001f46d289183b24bc65c"} Nov 03 19:40:30.088014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.080139 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8012d09a710ccfa438f45583d611075f5121c21ee27001f46d289183b24bc65c" Nov 03 19:40:30.088014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.068339 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn\" (UniqueName: \"kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn\") pod \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " Nov 03 19:40:30.088014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.080249 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca\") pod \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " Nov 03 19:40:30.088014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.080377 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca\") pod \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\" (UID: \"2e5dcbd5-ee40-4876-a3c6-4a699dd848e8\") " Nov 03 19:40:30.088624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.088593 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca" pod "2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" (UID: "2e5dcbd5-ee40-4876-a3c6-4a699dd848e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:30.189063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.188612 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:30.218536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.218120 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn" pod "2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" (UID: "2e5dcbd5-ee40-4876-a3c6-4a699dd848e8"). InnerVolumeSpecName "kube-api-access-w9jdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:30.252995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.241191 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca" pod "2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" (UID: "2e5dcbd5-ee40-4876-a3c6-4a699dd848e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:30.320000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.308701 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jdn\" (UniqueName: \"kubernetes.io/projected/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-kube-api-access-w9jdn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:30.320000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.308741 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:30.558993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.484451 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:30.682035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.680574 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:30.774008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.760211 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:30.966999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.965126 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:40:30.966999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.966050 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:40:30.994107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.994067 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:30.995011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:30.994952 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:40:31.149993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:31.149894 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:40:31.635997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:31.629565 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:40:31.981118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:31.981088 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:32.101799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.101766 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca\") pod \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " Nov 03 19:40:32.102234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.102208 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm\" (UniqueName: \"kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm\") pod \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " Nov 03 19:40:32.102401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.102384 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca\") pod \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\" (UID: \"f726797a-52fd-44d8-b5a4-d9cd42446fd8\") " Nov 03 19:40:32.143673 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.143618 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca" pod "f726797a-52fd-44d8-b5a4-d9cd42446fd8" (UID: "f726797a-52fd-44d8-b5a4-d9cd42446fd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:32.184919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.184864 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm" pod "f726797a-52fd-44d8-b5a4-d9cd42446fd8" (UID: "f726797a-52fd-44d8-b5a4-d9cd42446fd8"). InnerVolumeSpecName "kube-api-access-2jcxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:32.185197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.185177 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca" pod "f726797a-52fd-44d8-b5a4-d9cd42446fd8" (UID: "f726797a-52fd-44d8-b5a4-d9cd42446fd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:32.225004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.203812 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:32.225004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.203844 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcxm\" (UniqueName: \"kubernetes.io/projected/f726797a-52fd-44d8-b5a4-d9cd42446fd8-kube-api-access-2jcxm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:32.225004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.203859 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726797a-52fd-44d8-b5a4-d9cd42446fd8-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:32.767788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.767746 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" Nov 03 19:40:32.770091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.770040 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72" event={"ID":"f726797a-52fd-44d8-b5a4-d9cd42446fd8","Type":"ContainerDied","Data":"a87f4f49e189d8e40f6fdbb83da85970eea82c3a803bca48dfdf518660181389"} Nov 03 19:40:32.770204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:32.770098 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87f4f49e189d8e40f6fdbb83da85970eea82c3a803bca48dfdf518660181389" Nov 03 19:40:33.258622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.258589 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86"] Nov 03 19:40:33.259192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.259178 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" containerName="revoke-keys" Nov 03 19:40:33.259241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.259193 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" containerName="revoke-keys" Nov 03 19:40:33.259241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.259207 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f726797a-52fd-44d8-b5a4-d9cd42446fd8" containerName="revoke-keys" Nov 03 19:40:33.259241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.259213 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f726797a-52fd-44d8-b5a4-d9cd42446fd8" containerName="revoke-keys" Nov 03 19:40:33.259332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.259310 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" containerName="revoke-keys" Nov 03 19:40:33.259332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.259318 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f726797a-52fd-44d8-b5a4-d9cd42446fd8" containerName="revoke-keys" Nov 03 19:40:33.262990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.262960 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.270337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.270255 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:40:33.270459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.270423 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:40:33.282270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.282242 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86"] Nov 03 19:40:33.355563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.355525 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.355756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.355644 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.355823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.355761 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.456495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.456464 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.456696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.456561 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.456696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.456614 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.457207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.457188 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.459076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.459051 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.469516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.469493 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5\") pod \"maas-api-revoke-sub-a72c5758-pcs86\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.577438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.577362 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:33.864279 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:33.864253 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39db2050_a067_48d8_89e3_5bb8b77f48d3.slice/crio-3b51399f8fdbc11a57ca171d510f0a27bd7b8f9cd49451c1112d6177ff0b09b5 WatchSource:0}: Error finding container 3b51399f8fdbc11a57ca171d510f0a27bd7b8f9cd49451c1112d6177ff0b09b5: Status 404 returned error can't find the container with id 3b51399f8fdbc11a57ca171d510f0a27bd7b8f9cd49451c1112d6177ff0b09b5 Nov 03 19:40:33.864401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:33.864299 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86"] Nov 03 19:40:34.786037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:34.786004 5233 generic.go:356] "Generic (PLEG): container finished" podID="39db2050-a067-48d8-89e3-5bb8b77f48d3" containerID="9db3dee9f90333b144a175a49b20aa73ae219b0c716320d5d05245b993cf6cf2" exitCode=0 Nov 03 19:40:34.786454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:34.786085 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" event={"ID":"39db2050-a067-48d8-89e3-5bb8b77f48d3","Type":"ContainerDied","Data":"9db3dee9f90333b144a175a49b20aa73ae219b0c716320d5d05245b993cf6cf2"} Nov 03 19:40:34.786454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:34.786187 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" event={"ID":"39db2050-a067-48d8-89e3-5bb8b77f48d3","Type":"ContainerStarted","Data":"3b51399f8fdbc11a57ca171d510f0a27bd7b8f9cd49451c1112d6177ff0b09b5"} Nov 03 19:40:35.953663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.953639 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:35.983776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.983749 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca\") pod \"39db2050-a067-48d8-89e3-5bb8b77f48d3\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " Nov 03 19:40:35.983898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.983780 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca\") pod \"39db2050-a067-48d8-89e3-5bb8b77f48d3\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " Nov 03 19:40:35.983898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.983821 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5\") pod \"39db2050-a067-48d8-89e3-5bb8b77f48d3\" (UID: \"39db2050-a067-48d8-89e3-5bb8b77f48d3\") " Nov 03 19:40:35.984154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.984128 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca" pod "39db2050-a067-48d8-89e3-5bb8b77f48d3" (UID: "39db2050-a067-48d8-89e3-5bb8b77f48d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:35.984437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.984416 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:35.986044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.986015 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5" pod "39db2050-a067-48d8-89e3-5bb8b77f48d3" (UID: "39db2050-a067-48d8-89e3-5bb8b77f48d3"). InnerVolumeSpecName "kube-api-access-glps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:35.986191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:35.986166 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca" pod "39db2050-a067-48d8-89e3-5bb8b77f48d3" (UID: "39db2050-a067-48d8-89e3-5bb8b77f48d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:36.085985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.085889 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39db2050-a067-48d8-89e3-5bb8b77f48d3-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:36.085985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.085915 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/39db2050-a067-48d8-89e3-5bb8b77f48d3-kube-api-access-glps5\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:36.722721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.722691 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx"] Nov 03 19:40:36.723312 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.723297 5233 cpu_manager.go:414] "containerMap: removing container" podUID="39db2050-a067-48d8-89e3-5bb8b77f48d3" containerName="revoke-keys" Nov 03 19:40:36.723390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.723313 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39db2050-a067-48d8-89e3-5bb8b77f48d3" containerName="revoke-keys" Nov 03 19:40:36.723460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.723412 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="39db2050-a067-48d8-89e3-5bb8b77f48d3" containerName="revoke-keys" Nov 03 19:40:36.727072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.727054 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.735564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.735540 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx"] Nov 03 19:40:36.795244 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.795203 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wv2\" (UniqueName: \"kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.795421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.795250 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.795529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.795502 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.803679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.803646 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" Nov 03 19:40:36.803842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.803686 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86" event={"ID":"39db2050-a067-48d8-89e3-5bb8b77f48d3","Type":"ContainerDied","Data":"3b51399f8fdbc11a57ca171d510f0a27bd7b8f9cd49451c1112d6177ff0b09b5"} Nov 03 19:40:36.803842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.803730 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b51399f8fdbc11a57ca171d510f0a27bd7b8f9cd49451c1112d6177ff0b09b5" Nov 03 19:40:36.897036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.896962 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.897205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.897071 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wv2\" (UniqueName: \"kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.897205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.897120 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.898326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.897763 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.899755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.899734 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:36.918158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:36.918111 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wv2\" (UniqueName: \"kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2\") pod \"maas-api-revoke-sub-3024c683-xhwvx\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:37.048856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:37.048772 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:37.335162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:37.335128 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx"] Nov 03 19:40:37.335282 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:37.335262 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280a04f6_20e2_44e6_8ffe_7a8e98262ecb.slice/crio-26d38762868ab2a6d9da057a4ac73c3cfe14960de6b7acf52f5fdb9137193a7a WatchSource:0}: Error finding container 26d38762868ab2a6d9da057a4ac73c3cfe14960de6b7acf52f5fdb9137193a7a: Status 404 returned error can't find the container with id 26d38762868ab2a6d9da057a4ac73c3cfe14960de6b7acf52f5fdb9137193a7a Nov 03 19:40:37.813068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:37.813034 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" event={"ID":"280a04f6-20e2-44e6-8ffe-7a8e98262ecb","Type":"ContainerStarted","Data":"faa347a103f0d427b226571354719fdcb2ddff044b15ebb72cc240b5543097a6"} Nov 03 19:40:37.813209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:37.813092 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" event={"ID":"280a04f6-20e2-44e6-8ffe-7a8e98262ecb","Type":"ContainerStarted","Data":"26d38762868ab2a6d9da057a4ac73c3cfe14960de6b7acf52f5fdb9137193a7a"} Nov 03 19:40:37.839464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:37.839415 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" podStartSLOduration=1.839402336 podStartE2EDuration="1.839402336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:37.835246744 +0000 UTC m=+804.176360230" watchObservedRunningTime="2026-11-03 19:40:37.839402336 +0000 UTC m=+804.180515821" Nov 03 19:40:38.823275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:38.823241 5233 generic.go:356] "Generic (PLEG): container finished" podID="280a04f6-20e2-44e6-8ffe-7a8e98262ecb" containerID="faa347a103f0d427b226571354719fdcb2ddff044b15ebb72cc240b5543097a6" exitCode=0 Nov 03 19:40:38.823757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:38.823326 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" event={"ID":"280a04f6-20e2-44e6-8ffe-7a8e98262ecb","Type":"ContainerDied","Data":"faa347a103f0d427b226571354719fdcb2ddff044b15ebb72cc240b5543097a6"} Nov 03 19:40:39.038120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.038078 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2"] Nov 03 19:40:39.042586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.042567 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.049825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.049795 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2"] Nov 03 19:40:39.051842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.051824 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"kube-root-ca.crt\"" Nov 03 19:40:39.051944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.051879 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"openshift-service-ca.crt\"" Nov 03 19:40:39.053238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.053215 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"distinct-w3-036184-kserve-self-signed-certs\"" Nov 03 19:40:39.053335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.053318 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-036184\"/\"default-dockercfg-zqxbt\"" Nov 03 19:40:39.125862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.125782 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.125862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.125838 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.126130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.125910 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlg8h\" (UniqueName: \"kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.126130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.125959 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.126130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.126119 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.126285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.126151 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.227658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.227621 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.227839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.227765 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.227839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.227806 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.227998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.227841 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlg8h\" (UniqueName: \"kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.227998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.227882 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.227998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.227987 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.228157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.228128 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.228211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.228193 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.228365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.228340 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.230442 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.230422 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.230442 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.230433 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.241091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.241067 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlg8h\" (UniqueName: \"kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h\") pod \"distinct-w3-036184-kserve-c7c86cf5d-677h2\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.357014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.356958 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:39.533078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.533044 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8"] Nov 03 19:40:39.546599 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.544745 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.574810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.562661 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8"] Nov 03 19:40:39.633350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633314 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633362 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkd4\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633506 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633542 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633626 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633761 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633792 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633815 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.633923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.633863 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.648281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.648259 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2"] Nov 03 19:40:39.654939 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:39.654900 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86065665_60a4_4bce_89b6_3979cfe52fe9.slice/crio-f8d4b1044c1ea6650892dbc287571777bea9ab3608ae55cfcf2d167494525572 WatchSource:0}: Error finding container f8d4b1044c1ea6650892dbc287571777bea9ab3608ae55cfcf2d167494525572: Status 404 returned error can't find the container with id f8d4b1044c1ea6650892dbc287571777bea9ab3608ae55cfcf2d167494525572 Nov 03 19:40:39.735461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735432 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735474 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkd4\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735540 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735566 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735593 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735723 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735754 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735778 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735801 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735832 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.735915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.735889 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.736134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.736117 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.736203 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.736183 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.736457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.736423 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.737880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.737858 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.738338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.738318 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.748243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.748217 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkd4\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.749187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.749169 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:39.837842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.837813 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" event={"ID":"86065665-60a4-4bce-89b6-3979cfe52fe9","Type":"ContainerStarted","Data":"6297e818c7a47e8d29fc99f24ad5aec9d33042f4291732483b153a24ec88ebc7"} Nov 03 19:40:39.838181 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.837855 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" event={"ID":"86065665-60a4-4bce-89b6-3979cfe52fe9","Type":"ContainerStarted","Data":"f8d4b1044c1ea6650892dbc287571777bea9ab3608ae55cfcf2d167494525572"} Nov 03 19:40:39.855074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.855018 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podStartSLOduration=0.85500036 podStartE2EDuration="855.00036ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:39.854354241 +0000 UTC m=+806.195467727" watchObservedRunningTime="2026-11-03 19:40:39.85500036 +0000 UTC m=+806.196113844" Nov 03 19:40:39.885959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.885912 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:40.001048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:39.996178 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:40.146837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.146798 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca\") pod \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " Nov 03 19:40:40.147047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.146957 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca\") pod \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " Nov 03 19:40:40.147047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.147028 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2\" (UniqueName: \"kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2\") pod \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\" (UID: \"280a04f6-20e2-44e6-8ffe-7a8e98262ecb\") " Nov 03 19:40:40.147383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.147353 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca" pod "280a04f6-20e2-44e6-8ffe-7a8e98262ecb" (UID: "280a04f6-20e2-44e6-8ffe-7a8e98262ecb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:40.147460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.147450 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.148915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.148888 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca" pod "280a04f6-20e2-44e6-8ffe-7a8e98262ecb" (UID: "280a04f6-20e2-44e6-8ffe-7a8e98262ecb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:40.148964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.148945 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2" pod "280a04f6-20e2-44e6-8ffe-7a8e98262ecb" (UID: "280a04f6-20e2-44e6-8ffe-7a8e98262ecb"). InnerVolumeSpecName "kube-api-access-95wv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:40.199327 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:40.199299 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a12753f_6e1b_459f_baae_40742e1b158d.slice/crio-17f7025566144136b4292df9a2fd15ca843d1c788f75cfa4abef2587330b2b1e WatchSource:0}: Error finding container 17f7025566144136b4292df9a2fd15ca843d1c788f75cfa4abef2587330b2b1e: Status 404 returned error can't find the container with id 17f7025566144136b4292df9a2fd15ca843d1c788f75cfa4abef2587330b2b1e Nov 03 19:40:40.201596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.201569 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8"] Nov 03 19:40:40.202741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.202702 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:40.202810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.202793 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:40.202863 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.202848 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:40.248804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.248776 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.248906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.248813 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wv2\" (UniqueName: \"kubernetes.io/projected/280a04f6-20e2-44e6-8ffe-7a8e98262ecb-kube-api-access-95wv2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.251308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.251115 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p"] Nov 03 19:40:40.251803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.251788 5233 cpu_manager.go:414] "containerMap: removing container" podUID="280a04f6-20e2-44e6-8ffe-7a8e98262ecb" containerName="revoke-keys" Nov 03 19:40:40.251875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.251803 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280a04f6-20e2-44e6-8ffe-7a8e98262ecb" containerName="revoke-keys" Nov 03 19:40:40.251935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.251923 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="280a04f6-20e2-44e6-8ffe-7a8e98262ecb" containerName="revoke-keys" Nov 03 19:40:40.262714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.262685 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.307230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.307202 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p"] Nov 03 19:40:40.350302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350263 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350318 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350401 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350469 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350501 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350536 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795dz\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350659 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350785 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.350894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.350875 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.357622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.357602 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:40.454403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454370 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454478 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454520 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454562 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454596 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454629 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454664 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795dz\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454750 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454860 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.454952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.454885 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.455231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.455208 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.456104 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.455958 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.456291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.456265 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.456353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.456305 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.457333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.457305 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.457725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.457706 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.468334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.468310 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795dz\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.468649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.468633 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.580992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.580892 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:40.844086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.843998 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" event={"ID":"3a12753f-6e1b-459f-baae-40742e1b158d","Type":"ContainerStarted","Data":"57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f"} Nov 03 19:40:40.844086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.844065 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" event={"ID":"3a12753f-6e1b-459f-baae-40742e1b158d","Type":"ContainerStarted","Data":"17f7025566144136b4292df9a2fd15ca843d1c788f75cfa4abef2587330b2b1e"} Nov 03 19:40:40.845789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.845766 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" Nov 03 19:40:40.845910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.845806 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx" event={"ID":"280a04f6-20e2-44e6-8ffe-7a8e98262ecb","Type":"ContainerDied","Data":"26d38762868ab2a6d9da057a4ac73c3cfe14960de6b7acf52f5fdb9137193a7a"} Nov 03 19:40:40.845910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.845848 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d38762868ab2a6d9da057a4ac73c3cfe14960de6b7acf52f5fdb9137193a7a" Nov 03 19:40:40.876494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.876446 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podStartSLOduration=1.876423271 podStartE2EDuration="1.876423271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:40.873179394 +0000 UTC m=+807.214292882" watchObservedRunningTime="2026-11-03 19:40:40.876423271 +0000 UTC m=+807.217536757" Nov 03 19:40:40.882097 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:40.882073 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc83e3e7_a366_46d2_b351_7b8f23ffc5a1.slice/crio-41819f01931a9f03c60fd7f72f7e88963ccb79be441c295f77fb8cfd4de0a511 WatchSource:0}: Error finding container 41819f01931a9f03c60fd7f72f7e88963ccb79be441c295f77fb8cfd4de0a511: Status 404 returned error can't find the container with id 41819f01931a9f03c60fd7f72f7e88963ccb79be441c295f77fb8cfd4de0a511 Nov 03 19:40:40.882473 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.882453 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p"] Nov 03 19:40:40.884438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.884410 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:40.884525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.884477 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:40.884593 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.884522 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:40:40.886520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:40.886500 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:41.863862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:41.863822 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" event={"ID":"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1","Type":"ContainerStarted","Data":"f84ee0b87f868f3e873d24b5ef07da0f374e354210716f00c2481067215a5704"} Nov 03 19:40:41.864396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:41.863872 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" event={"ID":"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1","Type":"ContainerStarted","Data":"41819f01931a9f03c60fd7f72f7e88963ccb79be441c295f77fb8cfd4de0a511"} Nov 03 19:40:41.917000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:41.915064 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:41.917000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:41.915129 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:41.917000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:41.915164 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podStartSLOduration=1.9151445489999999 podStartE2EDuration="1.915144549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:41.898668467 +0000 UTC m=+808.239781951" watchObservedRunningTime="2026-11-03 19:40:41.915144549 +0000 UTC m=+808.256258035" Nov 03 19:40:42.582865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:42.582819 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:42.584303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:42.584270 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:42.584454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:42.584321 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:42.906050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:42.893294 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:42.906050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:42.893350 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:43.582163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.582043 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:43.582163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.582121 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:43.783914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.782254 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg"] Nov 03 19:40:43.789303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.786726 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:43.878906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.800248 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-435bd8\"/\"distinct-w2-435bd8-kserve-self-signed-certs\"" Nov 03 19:40:43.882395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.882365 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg"] Nov 03 19:40:43.894011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.889315 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:43.898070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.898037 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:43.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.960214 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:43.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.960260 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:43.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.960332 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:43.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.960496 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxq9x\" (UniqueName: \"kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:43.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.960555 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:43.963350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:43.960586 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.069024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.061898 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxq9x\" (UniqueName: \"kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.069024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.062061 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.069024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.062100 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.069024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.062181 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.069024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.062206 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.069024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.062348 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.096087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.091759 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.096087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.091775 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.096087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.092097 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.192055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.186001 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxq9x\" (UniqueName: \"kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.208992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.198715 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.231994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.212110 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs\") pod \"distinct-w2-435bd8-kserve-756fd644fc-p6zqg\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.299395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.299353 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:44.590990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.590939 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:44.591267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.591236 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:44.964668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.956103 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:44.964668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:44.956173 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:45.409995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:45.409473 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg"] Nov 03 19:40:45.465010 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:45.463609 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod319cda80_14b5_4529_8cc6_298f6b2e0353.slice/crio-b5f0a9fbcc00d4ee3601d3ffe44b9af456bb31c3a01a6936d7b937ef4bbb7730 WatchSource:0}: Error finding container b5f0a9fbcc00d4ee3601d3ffe44b9af456bb31c3a01a6936d7b937ef4bbb7730: Status 404 returned error can't find the container with id b5f0a9fbcc00d4ee3601d3ffe44b9af456bb31c3a01a6936d7b937ef4bbb7730 Nov 03 19:40:45.588128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:45.587653 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:45.588128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:45.587712 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:45.899996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:45.899322 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:45.899996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:45.899382 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:46.612365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.584424 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:46.612365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.584499 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:46.632446 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.631352 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:46.704670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.704113 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:40:46.710140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.709771 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:40:46.776598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.776546 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" event={"ID":"319cda80-14b5-4529-8cc6-298f6b2e0353","Type":"ContainerStarted","Data":"b5f0a9fbcc00d4ee3601d3ffe44b9af456bb31c3a01a6936d7b937ef4bbb7730"} Nov 03 19:40:46.901993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.901118 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:46.908990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:46.907042 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:47.583991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.583304 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:47.583991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.583366 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:47.805997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.789379 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" event={"ID":"319cda80-14b5-4529-8cc6-298f6b2e0353","Type":"ContainerStarted","Data":"8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18"} Nov 03 19:40:47.901264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.889330 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:40:47.901264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.889387 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:40:47.939262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.939197 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podStartSLOduration=4.9391776 podStartE2EDuration="4.9391776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:47.865888947 +0000 UTC m=+814.207002433" watchObservedRunningTime="2026-11-03 19:40:47.9391776 +0000 UTC m=+814.280291087" Nov 03 19:40:47.945635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.945607 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs"] Nov 03 19:40:47.951084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.951058 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:47.957252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.957228 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:40:47.957381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.957256 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs"] Nov 03 19:40:47.957563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.957541 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:40:47.983296 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.983266 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhm7p\" (UniqueName: \"kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:47.983296 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.983300 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:47.983499 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:47.983432 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.084157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.084112 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhm7p\" (UniqueName: \"kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.084327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.084162 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.084327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.084286 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.093211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.085071 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.124991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.119745 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.124991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.122632 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhm7p\" (UniqueName: \"kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p\") pod \"maas-api-revoke-sub-4d38c760-ht9bs\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.272682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.272651 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:48.305992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.305074 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:48.581633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.581533 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:40:48.581633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.581601 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 19:40:48.610778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.610645 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs"] Nov 03 19:40:48.612367 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:40:48.612339 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a162266_d513_4951_9a33_ddfadd231667.slice/crio-0a8e275c5258838fdada115ebc624354535277551e983927237845bbcdf08f7a WatchSource:0}: Error finding container 0a8e275c5258838fdada115ebc624354535277551e983927237845bbcdf08f7a: Status 404 returned error can't find the container with id 0a8e275c5258838fdada115ebc624354535277551e983927237845bbcdf08f7a Nov 03 19:40:48.800481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:48.800446 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" event={"ID":"9a162266-d513-4951-9a33-ddfadd231667","Type":"ContainerStarted","Data":"0a8e275c5258838fdada115ebc624354535277551e983927237845bbcdf08f7a"} Nov 03 19:40:49.359042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.357996 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:49.377854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.377550 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:49.488716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.488322 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:49.489628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.489266 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:49.491517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.491278 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:40:49.585210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.585182 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:49.585712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.585684 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:49.586450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.586428 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:40:49.809303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.809273 5233 generic.go:356] "Generic (PLEG): container finished" podID="9a162266-d513-4951-9a33-ddfadd231667" containerID="324ab5f150ab4cb10d8b4840253cae4dfa86687f47767e7c7f5fd8f9067e880f" exitCode=0 Nov 03 19:40:49.809522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.809362 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" event={"ID":"9a162266-d513-4951-9a33-ddfadd231667","Type":"ContainerDied","Data":"324ab5f150ab4cb10d8b4840253cae4dfa86687f47767e7c7f5fd8f9067e880f"} Nov 03 19:40:49.820931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:49.820905 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:40:50.982492 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:50.982467 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:51.057093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.057063 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca\") pod \"9a162266-d513-4951-9a33-ddfadd231667\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " Nov 03 19:40:51.057283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.057271 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p\" (UniqueName: \"kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p\") pod \"9a162266-d513-4951-9a33-ddfadd231667\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " Nov 03 19:40:51.057349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.057336 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca\") pod \"9a162266-d513-4951-9a33-ddfadd231667\" (UID: \"9a162266-d513-4951-9a33-ddfadd231667\") " Nov 03 19:40:51.057746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.057713 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca" pod "9a162266-d513-4951-9a33-ddfadd231667" (UID: "9a162266-d513-4951-9a33-ddfadd231667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:51.059438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.059409 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p" pod "9a162266-d513-4951-9a33-ddfadd231667" (UID: "9a162266-d513-4951-9a33-ddfadd231667"). InnerVolumeSpecName "kube-api-access-dhm7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:51.059438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.059412 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca" pod "9a162266-d513-4951-9a33-ddfadd231667" (UID: "9a162266-d513-4951-9a33-ddfadd231667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:51.159429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.159345 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhm7p\" (UniqueName: \"kubernetes.io/projected/9a162266-d513-4951-9a33-ddfadd231667-kube-api-access-dhm7p\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:51.159429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.159376 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a162266-d513-4951-9a33-ddfadd231667-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:51.159429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.159390 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a162266-d513-4951-9a33-ddfadd231667-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:51.277875 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:40:51.277839 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:40:51.833430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.833402 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" Nov 03 19:40:51.841949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.841909 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs" event={"ID":"9a162266-d513-4951-9a33-ddfadd231667","Type":"ContainerDied","Data":"0a8e275c5258838fdada115ebc624354535277551e983927237845bbcdf08f7a"} Nov 03 19:40:51.842142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:51.841984 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8e275c5258838fdada115ebc624354535277551e983927237845bbcdf08f7a" Nov 03 19:40:54.302303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:54.302274 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:54.315396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:54.315372 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:40:54.866935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:40:54.866907 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:41:00.056119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.056084 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh"] Nov 03 19:41:00.056741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.056725 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9a162266-d513-4951-9a33-ddfadd231667" containerName="revoke-keys" Nov 03 19:41:00.056792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.056742 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a162266-d513-4951-9a33-ddfadd231667" containerName="revoke-keys" Nov 03 19:41:00.056869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.056860 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a162266-d513-4951-9a33-ddfadd231667" containerName="revoke-keys" Nov 03 19:41:00.060501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.060485 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.068708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.068687 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c5f6cb\"/\"distinct-w6-c5f6cb-kserve-self-signed-certs\"" Nov 03 19:41:00.074157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.074135 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh"] Nov 03 19:41:00.124663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.124622 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.124857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.124740 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jwl\" (UniqueName: \"kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.124857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.124784 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.124953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.124865 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.124953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.124884 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.124953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.124927 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.226874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.226825 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.226874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.226888 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.227254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.226946 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.227867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.227840 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.228053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.227850 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.228314 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.228251 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.228406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.228334 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jwl\" (UniqueName: \"kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.228406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.228377 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.228718 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.228696 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.231371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.231343 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.233592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.233569 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.242178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.242151 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jwl\" (UniqueName: \"kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl\") pod \"distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.374704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.374607 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:00.665726 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:00.665699 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd199580a_2d32_4e95_bb54_efe923ce3397.slice/crio-71098cc85171e825d791c435d177d77974047971892b4e8dabfb51e0b7de2f6c WatchSource:0}: Error finding container 71098cc85171e825d791c435d177d77974047971892b4e8dabfb51e0b7de2f6c: Status 404 returned error can't find the container with id 71098cc85171e825d791c435d177d77974047971892b4e8dabfb51e0b7de2f6c Nov 03 19:41:00.666092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.666067 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh"] Nov 03 19:41:00.954518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.954436 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" event={"ID":"d199580a-2d32-4e95-bb54-efe923ce3397","Type":"ContainerStarted","Data":"138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b"} Nov 03 19:41:00.954518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.954482 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" event={"ID":"d199580a-2d32-4e95-bb54-efe923ce3397","Type":"ContainerStarted","Data":"71098cc85171e825d791c435d177d77974047971892b4e8dabfb51e0b7de2f6c"} Nov 03 19:41:00.976285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:00.976233 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podStartSLOduration=0.976212982 podStartE2EDuration="976.212982ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:00.97465193 +0000 UTC m=+827.315765416" watchObservedRunningTime="2026-11-03 19:41:00.976212982 +0000 UTC m=+827.317326468" Nov 03 19:41:01.375311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:01.375271 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:09.967581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:09.967546 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8"] Nov 03 19:41:09.973080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:09.973058 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:09.989959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:09.989931 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8"] Nov 03 19:41:10.068240 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.068186 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb"] Nov 03 19:41:10.072747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.072725 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.080437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.080400 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.080613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.080444 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.080613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.080525 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.081626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.080634 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.081761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.081699 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.081761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.081742 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zbd\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.081866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.081844 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.081920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.081903 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.082017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.081944 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.107018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.106755 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb"] Nov 03 19:41:10.183540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183507 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.183540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183541 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.183754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183562 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.183754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183688 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.183754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183726 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.183754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183743 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.183900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183825 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.183900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183844 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.183900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.183886 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184036 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184066 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184129 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48q5\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.184202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184189 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184232 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184307 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184334 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zbd\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184363 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184371 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.184565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184408 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184485 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.184565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184507 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184542 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.184736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.184650 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.186481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.186436 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.186758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.186735 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.197634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.197604 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.198657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.198557 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zbd\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.285647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285558 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.285647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285598 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.285647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285624 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.285907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285683 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.285907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285707 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.285907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285751 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.285903 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48q5\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286047 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286097 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286101 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286154 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286418 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286558 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286519 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.286668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.286647 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.288285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.288255 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.288436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.288417 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.295353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.295330 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:10.302485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.302462 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.302685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.302665 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48q5\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.375749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.375708 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:10.397415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.396153 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:10.397415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.397316 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:10.596025 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:10.595620 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb36251e_85de_4139_ac16_d0642a2bbace.slice/crio-32502ee6c964326d063a45368547b90c828340ac90ddfa982a596deb4c538c42 WatchSource:0}: Error finding container 32502ee6c964326d063a45368547b90c828340ac90ddfa982a596deb4c538c42: Status 404 returned error can't find the container with id 32502ee6c964326d063a45368547b90c828340ac90ddfa982a596deb4c538c42 Nov 03 19:41:10.598363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.598311 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:10.598471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.598393 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:10.598471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.598422 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8"] Nov 03 19:41:10.598471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.598441 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:10.714992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.714954 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb"] Nov 03 19:41:10.719485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.719441 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:10.719585 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.719544 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:10.719646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:10.719590 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:11.162851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.162798 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" event={"ID":"b5d17247-e25a-435e-9310-900d1fd65797","Type":"ContainerStarted","Data":"b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691"} Nov 03 19:41:11.163462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.163417 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" event={"ID":"b5d17247-e25a-435e-9310-900d1fd65797","Type":"ContainerStarted","Data":"89a04f081f7e67b7e50dc44b06a0333c7eb9880e26ecab39e00cee7bbc82b5ef"} Nov 03 19:41:11.186815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.186696 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" event={"ID":"cb36251e-85de-4139-ac16-d0642a2bbace","Type":"ContainerStarted","Data":"3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d"} Nov 03 19:41:11.187005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.186881 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" event={"ID":"cb36251e-85de-4139-ac16-d0642a2bbace","Type":"ContainerStarted","Data":"32502ee6c964326d063a45368547b90c828340ac90ddfa982a596deb4c538c42"} Nov 03 19:41:11.211905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.203320 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podStartSLOduration=1.203299272 podStartE2EDuration="1.203299272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:11.188845411 +0000 UTC m=+837.529958891" watchObservedRunningTime="2026-11-03 19:41:11.203299272 +0000 UTC m=+837.544412758" Nov 03 19:41:11.260007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.256107 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:11.273262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.273205 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podStartSLOduration=2.27318772 podStartE2EDuration="2.27318772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:11.230775891 +0000 UTC m=+837.571889378" watchObservedRunningTime="2026-11-03 19:41:11.27318772 +0000 UTC m=+837.614301207" Nov 03 19:41:11.307144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.307023 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:11.314211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.314171 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:11.314457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.314428 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:11.448052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.442198 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:11.553047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.530006 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:11.553282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:11.553258 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:12.314771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.314736 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:12.319093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.319050 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:12.405995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.400412 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:12.405995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.400474 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:12.662581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.662061 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:12.821137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.821084 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:12.899032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.888240 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:12.899032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:12.889235 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:13.298712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.298294 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:13.298712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.298353 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:13.398900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.398862 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:13.399523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.399491 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:13.634052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.633210 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:13.676996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.667876 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:13.676996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.671083 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:41:13.703993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.703288 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:13.842996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.823054 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:13.842996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.823807 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:41:14.039242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.970475 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:41:14.039242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:13.970961 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:41:14.062029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.041716 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:41:14.342058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.329479 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:14.342058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.329535 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:14.559940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.559904 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:14.631169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.605466 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:14.690992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.641165 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:14.690992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.641229 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:14.690992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.688221 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:14.814995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.790004 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:41:14.818444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.806437 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:14.955124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.864952 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:41:14.955804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.955686 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:41:14.977171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:14.879086 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:15.074994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.074800 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:15.079011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.075946 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:41:15.079529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.078801 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:15.084298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.084280 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:41:15.279381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.277394 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm"] Nov 03 19:41:15.288474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.288450 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.313992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.298306 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:15.313992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.299153 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:15.313992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.305043 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:41:15.313992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.305700 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:41:15.339221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.339068 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:15.386170 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.385105 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:41:15.400038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.398271 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkdt\" (UniqueName: \"kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.400038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.398340 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.400038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.398537 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.400038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.399687 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:15.400038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.399729 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:15.509366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.504410 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.509366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.504586 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkdt\" (UniqueName: \"kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.509366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.504663 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.509366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.506767 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.615829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.615713 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkdt\" (UniqueName: \"kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.619331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.618635 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:41:15.638999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.633752 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bacc55f-8dbmm\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.767195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.767160 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm"] Nov 03 19:41:15.870180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.870095 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:15.974045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:15.973271 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:41:16.076841 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.076245 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:41:16.076841 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.076301 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:41:16.341138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.330153 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:16.341138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.330209 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:16.462380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.462292 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:16.462651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.462613 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:16.745779 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:16.745731 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bb9b9f_e098_402b_b426_31fdac25a95c.slice/crio-ffa056765c65d209dc33a5bd075b5c875af7f0d2a3929bf4e99a97854885c253 WatchSource:0}: Error finding container ffa056765c65d209dc33a5bd075b5c875af7f0d2a3929bf4e99a97854885c253: Status 404 returned error can't find the container with id ffa056765c65d209dc33a5bd075b5c875af7f0d2a3929bf4e99a97854885c253 Nov 03 19:41:16.750521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:16.750496 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm"] Nov 03 19:41:17.302993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.301323 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:17.302993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.301384 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:17.406047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.405561 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" event={"ID":"c1bb9b9f-e098-402b-b426-31fdac25a95c","Type":"ContainerStarted","Data":"bd9f2e9b818403b7dfa0137a374fdc0591cb1b6a6567b71890d21e20326ec53e"} Nov 03 19:41:17.406047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.405647 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" event={"ID":"c1bb9b9f-e098-402b-b426-31fdac25a95c","Type":"ContainerStarted","Data":"ffa056765c65d209dc33a5bd075b5c875af7f0d2a3929bf4e99a97854885c253"} Nov 03 19:41:17.410992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.407405 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:17.410992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.407478 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:17.455995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:17.455672 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" podStartSLOduration=3.455638461 podStartE2EDuration="3.455638461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:17.436357427 +0000 UTC m=+843.777470913" watchObservedRunningTime="2026-11-03 19:41:17.455638461 +0000 UTC m=+843.796751950" Nov 03 19:41:18.028686 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:41:18.028652 5233 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bb9b9f_e098_402b_b426_31fdac25a95c.slice/crio-bd9f2e9b818403b7dfa0137a374fdc0591cb1b6a6567b71890d21e20326ec53e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:41:18.296741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:18.296647 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:41:18.296741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:18.296718 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:41:18.399344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:18.399304 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:41:18.399521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:18.399364 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:41:18.415418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:18.415387 5233 generic.go:356] "Generic (PLEG): container finished" podID="c1bb9b9f-e098-402b-b426-31fdac25a95c" containerID="bd9f2e9b818403b7dfa0137a374fdc0591cb1b6a6567b71890d21e20326ec53e" exitCode=0 Nov 03 19:41:18.415641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:18.415624 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" event={"ID":"c1bb9b9f-e098-402b-b426-31fdac25a95c","Type":"ContainerDied","Data":"bd9f2e9b818403b7dfa0137a374fdc0591cb1b6a6567b71890d21e20326ec53e"} Nov 03 19:41:19.300125 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.300092 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:19.300763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.300738 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:19.301384 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.301361 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:41:19.659878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.659851 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:19.723232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.723181 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:19.724302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.724274 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:19.725930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.725906 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:41:19.798004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.796476 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca\") pod \"c1bb9b9f-e098-402b-b426-31fdac25a95c\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " Nov 03 19:41:19.798004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.796680 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt\" (UniqueName: \"kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt\") pod \"c1bb9b9f-e098-402b-b426-31fdac25a95c\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " Nov 03 19:41:19.798004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.796745 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca\") pod \"c1bb9b9f-e098-402b-b426-31fdac25a95c\" (UID: \"c1bb9b9f-e098-402b-b426-31fdac25a95c\") " Nov 03 19:41:19.820517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.810211 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca" pod "c1bb9b9f-e098-402b-b426-31fdac25a95c" (UID: "c1bb9b9f-e098-402b-b426-31fdac25a95c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:19.820937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.820883 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt" pod "c1bb9b9f-e098-402b-b426-31fdac25a95c" (UID: "c1bb9b9f-e098-402b-b426-31fdac25a95c"). InnerVolumeSpecName "kube-api-access-4gkdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:19.833071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.825146 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca" pod "c1bb9b9f-e098-402b-b426-31fdac25a95c" (UID: "c1bb9b9f-e098-402b-b426-31fdac25a95c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:19.900171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.900101 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkdt\" (UniqueName: \"kubernetes.io/projected/c1bb9b9f-e098-402b-b426-31fdac25a95c-kube-api-access-4gkdt\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:19.900171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.900132 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:19.900171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:19.900147 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1bb9b9f-e098-402b-b426-31fdac25a95c-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:20.454138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.454104 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" event={"ID":"c1bb9b9f-e098-402b-b426-31fdac25a95c","Type":"ContainerDied","Data":"ffa056765c65d209dc33a5bd075b5c875af7f0d2a3929bf4e99a97854885c253"} Nov 03 19:41:20.454946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.454162 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa056765c65d209dc33a5bd075b5c875af7f0d2a3929bf4e99a97854885c253" Nov 03 19:41:20.454946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.454113 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm" Nov 03 19:41:20.845504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.844420 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz"] Nov 03 19:41:20.845504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.845440 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c1bb9b9f-e098-402b-b426-31fdac25a95c" containerName="revoke-keys" Nov 03 19:41:20.845504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.845471 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bb9b9f-e098-402b-b426-31fdac25a95c" containerName="revoke-keys" Nov 03 19:41:20.845826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.845712 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bb9b9f-e098-402b-b426-31fdac25a95c" containerName="revoke-keys" Nov 03 19:41:20.849651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.849618 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:20.861772 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.861737 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz"] Nov 03 19:41:20.865938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.862359 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 19:41:20.865938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.862488 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 19:41:20.865938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.865602 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 19:41:20.867930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:20.866598 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8qmjp\"" Nov 03 19:41:21.016880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.016852 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.017097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.016899 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.017283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.017256 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grst\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.017360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.017320 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.017443 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.017420 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.017504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.017462 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.017550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.017517 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.118571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118471 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.118739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118619 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grst\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.118739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118678 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.118739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118712 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.118739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118737 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.119018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118789 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.119018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.118932 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.119577 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.119547 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.121230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.121185 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.121396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.121376 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.121727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.121698 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.121831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.121740 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.135055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.135034 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.135160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.135137 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grst\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tpjzz\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.187173 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.187144 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:21.294007 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:41:21.292169 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:41:21.503267 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:21.503208 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef902ec_6022_4a16_be76_070d6d3f0f65.slice/crio-8c538bb30c005f9f91f86cfd550d8e77147ee02204b05d245f5690f1a935b5ab WatchSource:0}: Error finding container 8c538bb30c005f9f91f86cfd550d8e77147ee02204b05d245f5690f1a935b5ab: Status 404 returned error can't find the container with id 8c538bb30c005f9f91f86cfd550d8e77147ee02204b05d245f5690f1a935b5ab Nov 03 19:41:21.504099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:21.503946 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz"] Nov 03 19:41:22.470335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:22.470302 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" event={"ID":"aef902ec-6022-4a16-be76-070d6d3f0f65","Type":"ContainerStarted","Data":"8c538bb30c005f9f91f86cfd550d8e77147ee02204b05d245f5690f1a935b5ab"} Nov 03 19:41:24.577127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.576933 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq"] Nov 03 19:41:24.601016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.600278 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.603212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.603182 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq"] Nov 03 19:41:24.640437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.637374 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w"] Nov 03 19:41:24.651783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.651275 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.658949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.658918 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w"] Nov 03 19:41:24.672805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.672769 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.672944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.672820 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.672944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.672851 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.672944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.672892 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.672944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.672922 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.672954 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673008 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673031 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jqcx\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673057 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673084 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673144 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673178 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673277 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673346 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673401 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673490 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6rk\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673527 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.673716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.673576 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.774457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774420 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6rk\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.774640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774468 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.774640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774496 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.774640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774628 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774659 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774692 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774754 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774797 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774842 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774873 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774887 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774915 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jqcx\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.774943 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775002 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775034 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775083 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775113 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775151 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775176 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775196 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775248 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775294 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775393 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775407 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.775923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775599 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.775923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.775723 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.776144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.776103 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.777262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.777225 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.777894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.777860 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.778041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.777923 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.778369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.778344 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.779021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.778997 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.788671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.788649 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.790094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.790053 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.790202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.790087 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jqcx\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.793146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.793104 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6rk\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk\") pod \"e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:24.876854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.876819 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:24.877002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.876891 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:24.946407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.946378 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:24.981316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:24.981295 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:25.263202 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:25.263163 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e68eb51_7d02_4b99_8ff0_49d6319aa7a5.slice/crio-03cea38257be6d0655319c03008cb85cbbb39e426e401bedf80722d892d1b1f7 WatchSource:0}: Error finding container 03cea38257be6d0655319c03008cb85cbbb39e426e401bedf80722d892d1b1f7: Status 404 returned error can't find the container with id 03cea38257be6d0655319c03008cb85cbbb39e426e401bedf80722d892d1b1f7 Nov 03 19:41:25.264082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.264058 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq"] Nov 03 19:41:25.266415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.266376 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:25.266506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.266459 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:25.266506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.266499 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:25.337676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.337629 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w"] Nov 03 19:41:25.343538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.342821 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:25.343538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.342910 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:25.343538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.342952 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:25.581755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.580494 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" event={"ID":"cb41c5be-5f14-441b-bdbc-2cd395167c5d","Type":"ContainerStarted","Data":"a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e"} Nov 03 19:41:25.581755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.580600 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" event={"ID":"cb41c5be-5f14-441b-bdbc-2cd395167c5d","Type":"ContainerStarted","Data":"d60308cccb862d53112db81cbded01468abb18f0468c8eabe0e9c33b8de99641"} Nov 03 19:41:25.584527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.584495 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" event={"ID":"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5","Type":"ContainerStarted","Data":"c55c8ab0739495269844374fbf760d0e78a1d58b130a4e7cc323798a0164f8a3"} Nov 03 19:41:25.584678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.584549 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" event={"ID":"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5","Type":"ContainerStarted","Data":"03cea38257be6d0655319c03008cb85cbbb39e426e401bedf80722d892d1b1f7"} Nov 03 19:41:25.588466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.587842 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" event={"ID":"aef902ec-6022-4a16-be76-070d6d3f0f65","Type":"ContainerStarted","Data":"1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30"} Nov 03 19:41:25.588466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.588273 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:25.625993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.622960 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podStartSLOduration=1.622939028 podStartE2EDuration="1.622939028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:25.604574964 +0000 UTC m=+851.945688444" watchObservedRunningTime="2026-11-03 19:41:25.622939028 +0000 UTC m=+851.964052515" Nov 03 19:41:25.636604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.635445 5233 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tpjzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:41:25.636604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.635500 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" podUID="aef902ec-6022-4a16-be76-070d6d3f0f65" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:41:25.670027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.669284 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podStartSLOduration=1.669261166 podStartE2EDuration="1.669261166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:25.660693611 +0000 UTC m=+852.001807110" watchObservedRunningTime="2026-11-03 19:41:25.669261166 +0000 UTC m=+852.010374653" Nov 03 19:41:25.701994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.697601 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" podStartSLOduration=2.327681865 podStartE2EDuration="5.697580592s" totalImagesPullingTime="3.369898727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:25.695332037 +0000 UTC m=+852.036445524" watchObservedRunningTime="2026-11-03 19:41:25.697580592 +0000 UTC m=+852.038694080" Nov 03 19:41:25.947403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.947311 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:25.982997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:25.981608 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:26.294995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.294483 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8"] Nov 03 19:41:26.362504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.360873 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8"] Nov 03 19:41:26.363596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.363175 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.377416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.377323 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:41:26.377588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.377468 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:41:26.460994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.458508 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.461331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.461268 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcc2\" (UniqueName: \"kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.481411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.465175 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.582943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.582861 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.582943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.582907 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcc2\" (UniqueName: \"kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.583398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.583079 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.584035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.584003 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.650465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.650429 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.653387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.652234 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcc2\" (UniqueName: \"kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2\") pod \"maas-api-revoke-sub-b7025055-vwdr8\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.672428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.670056 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:41:26.712214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.712181 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:26.957993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.948705 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:26.957993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:26.948780 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:27.016994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.014120 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:27.016994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.014194 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:27.561555 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:27.561520 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ca99ed_8832_494e_8979_a5b3d7e40be9.slice/crio-e2d216b5f4ede88727557d07b9878a2e7634c04b51e892a18d2569b257edb275 WatchSource:0}: Error finding container e2d216b5f4ede88727557d07b9878a2e7634c04b51e892a18d2569b257edb275: Status 404 returned error can't find the container with id e2d216b5f4ede88727557d07b9878a2e7634c04b51e892a18d2569b257edb275 Nov 03 19:41:27.565663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.565633 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8"] Nov 03 19:41:27.679894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.679538 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" event={"ID":"e1ca99ed-8832-494e-8979-a5b3d7e40be9","Type":"ContainerStarted","Data":"e2d216b5f4ede88727557d07b9878a2e7634c04b51e892a18d2569b257edb275"} Nov 03 19:41:27.948430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.948395 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:27.948684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.948654 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:27.982294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.982096 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:27.982294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:27.982142 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:28.728755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:28.728722 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" event={"ID":"e1ca99ed-8832-494e-8979-a5b3d7e40be9","Type":"ContainerStarted","Data":"64c7f008ddfa7aeddb5a161ce7eb1ade478cceaeb4a3c71fe0209143b25d10ca"} Nov 03 19:41:28.762608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:28.762550 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" podStartSLOduration=2.762536872 podStartE2EDuration="2.762536872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:28.758400855 +0000 UTC m=+855.099514341" watchObservedRunningTime="2026-11-03 19:41:28.762536872 +0000 UTC m=+855.103650357" Nov 03 19:41:28.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:28.948600 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:28.957994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:28.948665 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:28.982551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:28.982464 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:28.982551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:28.982509 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:29.800996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:29.799631 5233 generic.go:356] "Generic (PLEG): container finished" podID="e1ca99ed-8832-494e-8979-a5b3d7e40be9" containerID="64c7f008ddfa7aeddb5a161ce7eb1ade478cceaeb4a3c71fe0209143b25d10ca" exitCode=0 Nov 03 19:41:29.800996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:29.799962 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" event={"ID":"e1ca99ed-8832-494e-8979-a5b3d7e40be9","Type":"ContainerDied","Data":"64c7f008ddfa7aeddb5a161ce7eb1ade478cceaeb4a3c71fe0209143b25d10ca"} Nov 03 19:41:29.948994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:29.948407 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:29.948994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:29.948467 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:30.005993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.005153 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:30.005993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.005455 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:30.713997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.713437 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:30.903115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.903072 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:30.954005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.953076 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:30.954005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.953140 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:30.984797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.984703 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:30.985065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:30.985032 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:31.096704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.096641 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:31.183058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.170069 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:31.192424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.192279 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:41:31.410302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.410183 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:31.717994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.716076 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:31.717994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.716828 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:41:31.955048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.953131 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:31.955048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.953215 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:31.993731 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:31.993638 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:32.058076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.058020 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:32.135834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.104067 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:32.161799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.161746 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:41:32.207911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.175878 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:32.207911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.180201 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:41:32.207911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.206910 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:41:32.208645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.208617 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:32.563156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.557209 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:41:32.743999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.742920 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:32.829327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.829269 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:32.918998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.918937 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:32.963120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.963090 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:32.982065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.967048 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:33.004902 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:32.987643 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.032687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.032121 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:33.032687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.032191 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.033218 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.033182 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:33.051000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.042482 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:33.059028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.051197 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:33.133693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.126642 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.229455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.219086 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.229455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.229077 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.354137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.231053 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2\" (UniqueName: \"kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2\") pod \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " Nov 03 19:41:33.354137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.231156 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca\") pod \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " Nov 03 19:41:33.354137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.231237 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca\") pod \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\" (UID: \"e1ca99ed-8832-494e-8979-a5b3d7e40be9\") " Nov 03 19:41:33.355852 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.355786 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca" pod "e1ca99ed-8832-494e-8979-a5b3d7e40be9" (UID: "e1ca99ed-8832-494e-8979-a5b3d7e40be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:33.385272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.359548 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:33.387159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.387096 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca" pod "e1ca99ed-8832-494e-8979-a5b3d7e40be9" (UID: "e1ca99ed-8832-494e-8979-a5b3d7e40be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:33.407019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.397194 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2" pod "e1ca99ed-8832-494e-8979-a5b3d7e40be9" (UID: "e1ca99ed-8832-494e-8979-a5b3d7e40be9"). InnerVolumeSpecName "kube-api-access-dvcc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:33.464997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.464127 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvcc2\" (UniqueName: \"kubernetes.io/projected/e1ca99ed-8832-494e-8979-a5b3d7e40be9-kube-api-access-dvcc2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:33.464997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.464169 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ca99ed-8832-494e-8979-a5b3d7e40be9-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:33.658051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.579200 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.658051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.594435 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:41:33.715385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.715345 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.784048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.781354 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.874996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.871049 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.926049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.925936 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:33.964001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.963502 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:33.964001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:33.963559 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:34.033007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.031705 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:34.033007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.031770 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:34.146997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.136187 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:34.183020 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.182905 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:34.183290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.183246 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:34.208116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.189365 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:41:34.222304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.222273 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" event={"ID":"e1ca99ed-8832-494e-8979-a5b3d7e40be9","Type":"ContainerDied","Data":"e2d216b5f4ede88727557d07b9878a2e7634c04b51e892a18d2569b257edb275"} Nov 03 19:41:34.224498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.224473 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2d216b5f4ede88727557d07b9878a2e7634c04b51e892a18d2569b257edb275" Nov 03 19:41:34.224720 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.224706 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:41:34.246249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.230060 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:41:34.246249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.222192 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:41:34.246249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.205667 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8" Nov 03 19:41:34.246842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.246797 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:34.595999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.595044 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:34.625004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.617685 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:41:34.660066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.651133 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:41:34.667487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.667458 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:41:34.751613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.751553 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:41:34.754591 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.754562 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:41:34.782159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.782123 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:34.843576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.843546 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:41:34.859224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.859145 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:41:34.874225 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.874198 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:41:34.901765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.877275 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:41:34.948793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.947536 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:41:34.948793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.947590 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:41:34.982832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.982775 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:41:34.983381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:34.982850 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:41:36.988994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:36.988143 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:36.988994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:36.988210 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:37.035005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.034071 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:37.035005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.034137 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:37.557317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.556803 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:37.558540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.558444 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:37.560997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.559533 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:41:37.805992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.798706 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:37.805992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.799336 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:37.805992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:37.800525 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:41:40.270851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.270809 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb"] Nov 03 19:41:40.271865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.271844 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e1ca99ed-8832-494e-8979-a5b3d7e40be9" containerName="revoke-keys" Nov 03 19:41:40.271903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.271870 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ca99ed-8832-494e-8979-a5b3d7e40be9" containerName="revoke-keys" Nov 03 19:41:40.272136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.272118 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ca99ed-8832-494e-8979-a5b3d7e40be9" containerName="revoke-keys" Nov 03 19:41:40.277415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.277395 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.299993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.299912 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300335 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300386 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300432 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300481 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300526 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4rn\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300577 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300693 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.302027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.300768 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.401962 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402045 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402083 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402121 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402160 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4rn\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402223 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402467 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402519 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402607 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402568 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402794 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.402930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402834 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.403008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.402951 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.403233 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.403210 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.403337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.403264 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.404685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.404659 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.406698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.406676 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.419748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.419712 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb"] Nov 03 19:41:40.444988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.444947 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4rn\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.449405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.449345 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.541507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.541422 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66"] Nov 03 19:41:40.549322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.549297 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.596732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.596698 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66"] Nov 03 19:41:40.604856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.604796 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.604900 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.604949 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.605148 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.605292 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.605431 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.605474 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.605852 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.605830 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.606041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.606022 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvfl\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.614213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.614192 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:40.708738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.708703 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.708875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.708765 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.708875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.708851 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.709095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.708877 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvfl\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.709095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.708959 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.709095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.709014 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.709095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.709057 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.709485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.709175 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.709485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.709210 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.713538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.709987 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.713538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.710276 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.713538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.711294 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.713880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.713853 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.715307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.715258 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.717904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.717860 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.719735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.719694 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.731162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.731135 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.731263 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.731226 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvfl\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl\") pod \"e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.865930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.865851 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:40.942417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.942347 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb"] Nov 03 19:41:40.951695 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:40.951657 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf021696d_29c5_4bcb_9996_b5227f770013.slice/crio-94b90404a4541a3201b8ad35d167a4096beb09fde0ca0a151b6e38c6ef3743e5 WatchSource:0}: Error finding container 94b90404a4541a3201b8ad35d167a4096beb09fde0ca0a151b6e38c6ef3743e5: Status 404 returned error can't find the container with id 94b90404a4541a3201b8ad35d167a4096beb09fde0ca0a151b6e38c6ef3743e5 Nov 03 19:41:40.954580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.954541 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:40.954682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.954636 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:40.954739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:40.954682 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:41.170425 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:41.170398 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bac15b2_cd9a_4400_9c88_3efa632d65a4.slice/crio-444f493fdf13e7332c8a6d831ffcd7964e57ac9edd4665ebaa24d51b90d1b37b WatchSource:0}: Error finding container 444f493fdf13e7332c8a6d831ffcd7964e57ac9edd4665ebaa24d51b90d1b37b: Status 404 returned error can't find the container with id 444f493fdf13e7332c8a6d831ffcd7964e57ac9edd4665ebaa24d51b90d1b37b Nov 03 19:41:41.170696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.170676 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66"] Nov 03 19:41:41.173769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.173735 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:41.173860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.173811 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:41.173901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.173854 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:41.319938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.319894 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" event={"ID":"f021696d-29c5-4bcb-9996-b5227f770013","Type":"ContainerStarted","Data":"a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13"} Nov 03 19:41:41.320399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.320003 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" event={"ID":"f021696d-29c5-4bcb-9996-b5227f770013","Type":"ContainerStarted","Data":"94b90404a4541a3201b8ad35d167a4096beb09fde0ca0a151b6e38c6ef3743e5"} Nov 03 19:41:41.322239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.322149 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" event={"ID":"2bac15b2-cd9a-4400-9c88-3efa632d65a4","Type":"ContainerStarted","Data":"dcb8990081a840a9914ad6c2b6adf73caf91a4eb3b477b3efdec8f214090d069"} Nov 03 19:41:41.322239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.322209 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" event={"ID":"2bac15b2-cd9a-4400-9c88-3efa632d65a4","Type":"ContainerStarted","Data":"444f493fdf13e7332c8a6d831ffcd7964e57ac9edd4665ebaa24d51b90d1b37b"} Nov 03 19:41:41.349759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.349709 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podStartSLOduration=1.349694132 podStartE2EDuration="1.349694132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:41.346230403 +0000 UTC m=+867.687343887" watchObservedRunningTime="2026-11-03 19:41:41.349694132 +0000 UTC m=+867.690807617" Nov 03 19:41:41.378115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.378055 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podStartSLOduration=1.378036128 podStartE2EDuration="1.378036128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:41.377173901 +0000 UTC m=+867.718287386" watchObservedRunningTime="2026-11-03 19:41:41.378036128 +0000 UTC m=+867.719149614" Nov 03 19:41:41.630995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.628042 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:41.866375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:41.866348 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:42.619999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:42.619803 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:41:42.619999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:42.619880 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 19:41:42.867630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:42.867593 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:41:42.867802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:42.867665 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:41:43.623995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:43.623323 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:41:43.623995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:43.623380 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 19:41:43.866430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:43.866399 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:41:43.866601 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:43.866445 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:41:44.640629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:44.638001 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:41:44.640629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:44.638076 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 19:41:44.873858 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:44.873720 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:41:44.873858 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:44.873812 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:41:45.647762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:45.647725 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:41:45.650462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:45.648422 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 19:41:45.867126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:45.867067 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:41:45.867293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:45.867134 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:41:46.626992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:46.620299 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:41:46.626992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:46.620374 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 19:41:46.866226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:46.866189 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:41:46.866633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:46.866259 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:41:47.615083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:47.615052 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 19:41:47.615262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:47.615115 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 19:41:47.866642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:47.866537 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:41:47.866642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:47.866607 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:41:48.622105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:48.622070 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:48.623318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:48.623293 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:48.625200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:48.625172 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:41:48.871813 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:48.871778 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:48.872348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:48.872295 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:48.873100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:48.873080 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:41:49.140517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.140436 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926"] Nov 03 19:41:49.146194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.146169 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.150257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.150237 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:41:49.150564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.150549 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:41:49.151321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.151297 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926"] Nov 03 19:41:49.204214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.204188 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.204473 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.204453 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.204536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.204481 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.310234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.310157 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.310234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.310204 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.310483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.310287 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.312247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.311386 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.313736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.313706 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.325116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.325085 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5\") pod \"maas-api-revoke-sub-9c628595-x8926\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.465109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.465081 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:49.772354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:49.772325 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926"] Nov 03 19:41:49.774084 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:49.774051 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69be721_0fa2_40a2_bea8_a1aa06b99c3d.slice/crio-8192b487291ffe98d9e5270fab786e01364f6905bfccc751d5c3ba655ac2404f WatchSource:0}: Error finding container 8192b487291ffe98d9e5270fab786e01364f6905bfccc751d5c3ba655ac2404f: Status 404 returned error can't find the container with id 8192b487291ffe98d9e5270fab786e01364f6905bfccc751d5c3ba655ac2404f Nov 03 19:41:50.443650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:50.443562 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" event={"ID":"a69be721-0fa2-40a2-bea8-a1aa06b99c3d","Type":"ContainerStarted","Data":"5d773f312dc5b0d91ed88c492c9478d93aa85698fbd5a5efbc9043c46551a750"} Nov 03 19:41:50.443650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:50.443616 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" event={"ID":"a69be721-0fa2-40a2-bea8-a1aa06b99c3d","Type":"ContainerStarted","Data":"8192b487291ffe98d9e5270fab786e01364f6905bfccc751d5c3ba655ac2404f"} Nov 03 19:41:50.468308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:50.468264 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" podStartSLOduration=1.46825236 podStartE2EDuration="1.46825236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:50.463373792 +0000 UTC m=+876.804487271" watchObservedRunningTime="2026-11-03 19:41:50.46825236 +0000 UTC m=+876.809365891" Nov 03 19:41:51.276938 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:41:51.276904 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:41:51.497282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:51.497242 5233 generic.go:356] "Generic (PLEG): container finished" podID="a69be721-0fa2-40a2-bea8-a1aa06b99c3d" containerID="5d773f312dc5b0d91ed88c492c9478d93aa85698fbd5a5efbc9043c46551a750" exitCode=0 Nov 03 19:41:51.497950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:51.497915 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" event={"ID":"a69be721-0fa2-40a2-bea8-a1aa06b99c3d","Type":"ContainerDied","Data":"5d773f312dc5b0d91ed88c492c9478d93aa85698fbd5a5efbc9043c46551a750"} Nov 03 19:41:52.677642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.677616 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:52.771883 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.771797 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca\") pod \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " Nov 03 19:41:52.772096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.772078 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5\") pod \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " Nov 03 19:41:52.772230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.772212 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca\") pod \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\" (UID: \"a69be721-0fa2-40a2-bea8-a1aa06b99c3d\") " Nov 03 19:41:52.772309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.772231 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca" pod "a69be721-0fa2-40a2-bea8-a1aa06b99c3d" (UID: "a69be721-0fa2-40a2-bea8-a1aa06b99c3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:52.772811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.772784 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:52.774036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.774003 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5" pod "a69be721-0fa2-40a2-bea8-a1aa06b99c3d" (UID: "a69be721-0fa2-40a2-bea8-a1aa06b99c3d"). InnerVolumeSpecName "kube-api-access-d4dm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:52.774659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.774634 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca" pod "a69be721-0fa2-40a2-bea8-a1aa06b99c3d" (UID: "a69be721-0fa2-40a2-bea8-a1aa06b99c3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:52.874396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.874362 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4dm5\" (UniqueName: \"kubernetes.io/projected/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-kube-api-access-d4dm5\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:52.874396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:52.874395 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69be721-0fa2-40a2-bea8-a1aa06b99c3d-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:53.517279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.517240 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr"] Nov 03 19:41:53.518117 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.518091 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a69be721-0fa2-40a2-bea8-a1aa06b99c3d" containerName="revoke-keys" Nov 03 19:41:53.518117 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.518111 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69be721-0fa2-40a2-bea8-a1aa06b99c3d" containerName="revoke-keys" Nov 03 19:41:53.518324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.518261 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69be721-0fa2-40a2-bea8-a1aa06b99c3d" containerName="revoke-keys" Nov 03 19:41:53.527989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.521051 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" Nov 03 19:41:53.527989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.523384 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926" event={"ID":"a69be721-0fa2-40a2-bea8-a1aa06b99c3d","Type":"ContainerDied","Data":"8192b487291ffe98d9e5270fab786e01364f6905bfccc751d5c3ba655ac2404f"} Nov 03 19:41:53.527989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.523421 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8192b487291ffe98d9e5270fab786e01364f6905bfccc751d5c3ba655ac2404f" Nov 03 19:41:53.527989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.523497 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.527989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.526842 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr"] Nov 03 19:41:53.529430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.529411 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:41:53.529660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.529620 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:41:53.583889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.583859 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.584065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.584046 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.584113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.584086 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphdf\" (UniqueName: \"kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.685877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.685843 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.686387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.685892 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphdf\" (UniqueName: \"kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.686387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.686105 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.686965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.686940 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.688334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.688305 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.698601 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.698578 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphdf\" (UniqueName: \"kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf\") pod \"maas-api-revoke-sub-1713bcdb-8lrvr\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:53.851331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:53.851228 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:54.167633 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:54.167598 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bca63ef_8a3f_4f44_b21c_bd2d6a3bf712.slice/crio-ae24d191b7a7a2e8e6eddfa69ec6c737867d120fd2ca50a019c844d1b5dd926c WatchSource:0}: Error finding container ae24d191b7a7a2e8e6eddfa69ec6c737867d120fd2ca50a019c844d1b5dd926c: Status 404 returned error can't find the container with id ae24d191b7a7a2e8e6eddfa69ec6c737867d120fd2ca50a019c844d1b5dd926c Nov 03 19:41:54.168545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:54.168521 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr"] Nov 03 19:41:54.533565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:54.533531 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" event={"ID":"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712","Type":"ContainerStarted","Data":"907d2a12b9b2d7a046ff8d9739ec128ec37132a7e1b61102b92ed42139ae81c2"} Nov 03 19:41:54.533731 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:54.533586 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" event={"ID":"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712","Type":"ContainerStarted","Data":"ae24d191b7a7a2e8e6eddfa69ec6c737867d120fd2ca50a019c844d1b5dd926c"} Nov 03 19:41:54.562274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:54.562210 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" podStartSLOduration=1.562192053 podStartE2EDuration="1.562192053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:54.555360032 +0000 UTC m=+880.896473520" watchObservedRunningTime="2026-11-03 19:41:54.562192053 +0000 UTC m=+880.903305541" Nov 03 19:41:55.054277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.054245 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb"] Nov 03 19:41:55.059799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.059777 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.080560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.080526 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb"] Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.120988 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121037 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121174 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121234 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121286 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121384 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121418 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121478 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.126111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.121603 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4l2\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.223952 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224049 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224150 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224181 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224221 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224333 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4l2\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224374 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224398 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224532 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.224991 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.227991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.225279 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.232027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.231055 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.232027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.231491 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.232027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.231938 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.234436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.234117 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.234436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.234384 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.245840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.245780 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.246548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.246450 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4l2\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2\") pod \"e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.497559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.497517 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:55.544507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.544473 5233 generic.go:356] "Generic (PLEG): container finished" podID="7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" containerID="907d2a12b9b2d7a046ff8d9739ec128ec37132a7e1b61102b92ed42139ae81c2" exitCode=0 Nov 03 19:41:55.544676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.544548 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" event={"ID":"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712","Type":"ContainerDied","Data":"907d2a12b9b2d7a046ff8d9739ec128ec37132a7e1b61102b92ed42139ae81c2"} Nov 03 19:41:55.812685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.812650 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb"] Nov 03 19:41:55.813180 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:41:55.813154 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046e4b29_a8e2_4b02_8b84_df0d3ee25143.slice/crio-a99bde08b16b5fbc5a5d1abbf125b90feb9291e77d60e07aa2aebadf784030d3 WatchSource:0}: Error finding container a99bde08b16b5fbc5a5d1abbf125b90feb9291e77d60e07aa2aebadf784030d3: Status 404 returned error can't find the container with id a99bde08b16b5fbc5a5d1abbf125b90feb9291e77d60e07aa2aebadf784030d3 Nov 03 19:41:55.816482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.816449 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:55.816587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.816509 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:55.816587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:55.816541 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:41:56.596303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:56.595069 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" event={"ID":"046e4b29-a8e2-4b02-8b84-df0d3ee25143","Type":"ContainerStarted","Data":"234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7"} Nov 03 19:41:56.596303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:56.595132 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" event={"ID":"046e4b29-a8e2-4b02-8b84-df0d3ee25143","Type":"ContainerStarted","Data":"a99bde08b16b5fbc5a5d1abbf125b90feb9291e77d60e07aa2aebadf784030d3"} Nov 03 19:41:56.664059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:56.662479 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podStartSLOduration=1.662456739 podStartE2EDuration="1.662456739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:56.629724342 +0000 UTC m=+882.970837829" watchObservedRunningTime="2026-11-03 19:41:56.662456739 +0000 UTC m=+883.003570227" Nov 03 19:41:56.961644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:56.961607 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:57.073475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.073441 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca\") pod \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " Nov 03 19:41:57.073645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.073504 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca\") pod \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " Nov 03 19:41:57.073798 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.073780 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf\" (UniqueName: \"kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf\") pod \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\" (UID: \"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712\") " Nov 03 19:41:57.073869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.073846 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca" pod "7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" (UID: "7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:57.074337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.074308 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:57.075566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.075540 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca" pod "7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" (UID: "7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:57.075801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.075785 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf" pod "7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" (UID: "7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712"). InnerVolumeSpecName "kube-api-access-dphdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:57.175396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.175313 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphdf\" (UniqueName: \"kubernetes.io/projected/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-kube-api-access-dphdf\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:57.175396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.175343 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:57.513000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.510039 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:41:57.518990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.518557 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:41:57.518990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.518611 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:41:57.668017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.667379 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" Nov 03 19:41:57.668017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.667379 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr" event={"ID":"7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712","Type":"ContainerDied","Data":"ae24d191b7a7a2e8e6eddfa69ec6c737867d120fd2ca50a019c844d1b5dd926c"} Nov 03 19:41:57.668017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:57.667743 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae24d191b7a7a2e8e6eddfa69ec6c737867d120fd2ca50a019c844d1b5dd926c" Nov 03 19:41:58.503997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:58.503082 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:41:58.503997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:58.503142 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:41:59.500993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:59.500327 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:41:59.500993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:41:59.500397 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:00.502995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:00.502584 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:42:00.502995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:00.502654 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:01.254997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.251122 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:42:01.627048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.626219 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded" Nov 03 19:42:01.710000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.629443 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:01.710000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.636380 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:01.710000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.641878 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:42:01.710000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.642709 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:01.710000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.643024 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:01.771995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.768193 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:42:01.913691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.861361 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:42:01.913691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:01.894151 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:42:02.331007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.245326 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:02.331007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.268924 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:42:02.412001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.371223 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:42:02.666055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.525260 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:42:02.666055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.525323 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:02.666055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.639195 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:02.666055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.640092 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:42:02.668198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.667543 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:02.679063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.668446 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:42:02.748016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.736051 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:02.748016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.736741 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:42:02.770027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.769280 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:02.770027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.769706 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:42:03.005058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.937834 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 19:42:03.005058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.938168 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:42:03.005058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.938548 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:42:03.005058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:02.965375 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:42:03.034008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.033963 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:42:03.089129 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.089095 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:42:03.093097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.093045 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:03.300057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.266685 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:42:03.300057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.281255 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:03.300057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.299947 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:42:03.580011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.579906 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:42:03.580291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.580260 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:03.662997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.661465 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:42:03.669066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.669026 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:03.746207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.746167 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:03.778348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.778301 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:03.939176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.939071 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:03.978366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:03.971079 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:04.247188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.247144 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:42:04.247390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.247214 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:04.302995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.301282 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:04.302995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.301394 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:42:04.529050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.525095 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:42:04.545162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.545118 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:04.705042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.700825 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:42:04.750896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.750854 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:42:04.774772 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.774734 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:42:04.823076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.781264 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:42:04.823076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.817194 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:04.823076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.812763 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:42:04.959003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.941950 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn"] Nov 03 19:42:04.959003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.943063 5233 cpu_manager.go:414] "containerMap: removing container" podUID="7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" containerName="revoke-keys" Nov 03 19:42:04.959003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.943084 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" containerName="revoke-keys" Nov 03 19:42:04.959003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:04.943312 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" containerName="revoke-keys" Nov 03 19:42:05.029210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.029178 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.043050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.043021 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:42:05.043357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.043339 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:42:05.105036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.103879 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.105036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.103954 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.105036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.104056 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ts4\" (UniqueName: \"kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.247778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.221143 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn"] Nov 03 19:42:05.247778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.222437 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ts4\" (UniqueName: \"kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.247778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.222725 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.247778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.222764 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.247778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.223650 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.378046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.352791 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ts4\" (UniqueName: \"kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.378214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.352076 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 19:42:05.416659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.416615 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:42:05.437108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.437070 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:42:05.465249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.465219 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:42:05.477049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.476801 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:42:05.622930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.620650 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:42:05.633651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.633229 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67bfc8fa-tmhvn\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.635064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.633966 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:42:05.635383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.634029 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:42:05.636812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.636505 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:05.636950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.636925 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:42:05.653701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.653678 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:42:05.677929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.676917 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:05.696319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.693825 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:42:05.696319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.693894 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:05.704634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.704597 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:05.758043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.758016 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:42:05.760584 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.760561 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:42:05.762409 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:05.762390 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:42:06.143019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:06.142988 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn"] Nov 03 19:42:06.143787 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:06.143748 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b34deef_f3d3_4006_a93f_302c977f63aa.slice/crio-e82dab4d539c70951402b5fbbec880c599ae6a8a16edb7702e72d4e6a36c12b5 WatchSource:0}: Error finding container e82dab4d539c70951402b5fbbec880c599ae6a8a16edb7702e72d4e6a36c12b5: Status 404 returned error can't find the container with id e82dab4d539c70951402b5fbbec880c599ae6a8a16edb7702e72d4e6a36c12b5 Nov 03 19:42:06.865540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:06.865446 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:42:06.866749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:06.866672 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:42:06.868009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:06.867990 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:42:07.096010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:07.094511 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" event={"ID":"9b34deef-f3d3-4006-a93f-302c977f63aa","Type":"ContainerStarted","Data":"581e7a886e4cb112d2b6a04ca5efc29c147d730bc428517d1c6f68ebefe5b105"} Nov 03 19:42:07.096010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:07.094577 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" event={"ID":"9b34deef-f3d3-4006-a93f-302c977f63aa","Type":"ContainerStarted","Data":"e82dab4d539c70951402b5fbbec880c599ae6a8a16edb7702e72d4e6a36c12b5"} Nov 03 19:42:07.122222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:07.122102 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" podStartSLOduration=4.122087664 podStartE2EDuration="4.122087664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:07.121163637 +0000 UTC m=+893.462277135" watchObservedRunningTime="2026-11-03 19:42:07.122087664 +0000 UTC m=+893.463201148" Nov 03 19:42:08.121020 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:08.112228 5233 generic.go:356] "Generic (PLEG): container finished" podID="9b34deef-f3d3-4006-a93f-302c977f63aa" containerID="581e7a886e4cb112d2b6a04ca5efc29c147d730bc428517d1c6f68ebefe5b105" exitCode=0 Nov 03 19:42:08.121020 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:08.113182 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" event={"ID":"9b34deef-f3d3-4006-a93f-302c977f63aa","Type":"ContainerDied","Data":"581e7a886e4cb112d2b6a04ca5efc29c147d730bc428517d1c6f68ebefe5b105"} Nov 03 19:42:09.288680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.288657 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:09.449765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.449677 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4\" (UniqueName: \"kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4\") pod \"9b34deef-f3d3-4006-a93f-302c977f63aa\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " Nov 03 19:42:09.449765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.449736 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca\") pod \"9b34deef-f3d3-4006-a93f-302c977f63aa\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " Nov 03 19:42:09.450028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.449777 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca\") pod \"9b34deef-f3d3-4006-a93f-302c977f63aa\" (UID: \"9b34deef-f3d3-4006-a93f-302c977f63aa\") " Nov 03 19:42:09.450174 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.450148 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca" pod "9b34deef-f3d3-4006-a93f-302c977f63aa" (UID: "9b34deef-f3d3-4006-a93f-302c977f63aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:09.450417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.450402 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:09.452324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.452303 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca" pod "9b34deef-f3d3-4006-a93f-302c977f63aa" (UID: "9b34deef-f3d3-4006-a93f-302c977f63aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:09.452425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.452390 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4" pod "9b34deef-f3d3-4006-a93f-302c977f63aa" (UID: "9b34deef-f3d3-4006-a93f-302c977f63aa"). InnerVolumeSpecName "kube-api-access-l9ts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:09.552098 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.552065 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ts4\" (UniqueName: \"kubernetes.io/projected/9b34deef-f3d3-4006-a93f-302c977f63aa-kube-api-access-l9ts4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:09.552098 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:09.552093 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b34deef-f3d3-4006-a93f-302c977f63aa-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:10.137589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.137505 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" Nov 03 19:42:10.137589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.137538 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn" event={"ID":"9b34deef-f3d3-4006-a93f-302c977f63aa","Type":"ContainerDied","Data":"e82dab4d539c70951402b5fbbec880c599ae6a8a16edb7702e72d4e6a36c12b5"} Nov 03 19:42:10.137589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.137584 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e82dab4d539c70951402b5fbbec880c599ae6a8a16edb7702e72d4e6a36c12b5" Nov 03 19:42:10.927680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.927642 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv"] Nov 03 19:42:10.928704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.928684 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9b34deef-f3d3-4006-a93f-302c977f63aa" containerName="revoke-keys" Nov 03 19:42:10.928764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.928708 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b34deef-f3d3-4006-a93f-302c977f63aa" containerName="revoke-keys" Nov 03 19:42:10.928852 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.928842 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b34deef-f3d3-4006-a93f-302c977f63aa" containerName="revoke-keys" Nov 03 19:42:10.933625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.933596 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:10.937962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.937940 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv"] Nov 03 19:42:10.938653 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.938628 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:42:10.938775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:10.938715 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:42:11.072605 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.072573 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.072775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.072636 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.072830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.072786 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9gf\" (UniqueName: \"kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.174587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.174554 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.174750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.174634 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.174750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.174700 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9gf\" (UniqueName: \"kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.175435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.175412 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.176914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.176890 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.188725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.188667 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9gf\" (UniqueName: \"kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf\") pod \"maas-api-revoke-sub-1abf6a74-phxkv\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.250367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.250331 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:11.586871 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:11.586117 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00fa23d_8cc1_49df_b163_569fb54e26c8.slice/crio-a7b86baf0b1b8db6d45fbead98c2c729bb60aa82f4ad705f18670f4736866b0c WatchSource:0}: Error finding container a7b86baf0b1b8db6d45fbead98c2c729bb60aa82f4ad705f18670f4736866b0c: Status 404 returned error can't find the container with id a7b86baf0b1b8db6d45fbead98c2c729bb60aa82f4ad705f18670f4736866b0c Nov 03 19:42:11.587456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:11.587432 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv"] Nov 03 19:42:12.157532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:12.157439 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" event={"ID":"c00fa23d-8cc1-49df-b163-569fb54e26c8","Type":"ContainerStarted","Data":"8aa8840b88260d5096d35668e3a2a1f6e4fd21b391789c656ea8e72ad13237bc"} Nov 03 19:42:12.157532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:12.157508 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" event={"ID":"c00fa23d-8cc1-49df-b163-569fb54e26c8","Type":"ContainerStarted","Data":"a7b86baf0b1b8db6d45fbead98c2c729bb60aa82f4ad705f18670f4736866b0c"} Nov 03 19:42:12.179477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:12.179419 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" podStartSLOduration=2.179401981 podStartE2EDuration="2.179401981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:12.178410771 +0000 UTC m=+898.519524257" watchObservedRunningTime="2026-11-03 19:42:12.179401981 +0000 UTC m=+898.520515467" Nov 03 19:42:13.167895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:13.167859 5233 generic.go:356] "Generic (PLEG): container finished" podID="c00fa23d-8cc1-49df-b163-569fb54e26c8" containerID="8aa8840b88260d5096d35668e3a2a1f6e4fd21b391789c656ea8e72ad13237bc" exitCode=0 Nov 03 19:42:13.168305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:13.167943 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" event={"ID":"c00fa23d-8cc1-49df-b163-569fb54e26c8","Type":"ContainerDied","Data":"8aa8840b88260d5096d35668e3a2a1f6e4fd21b391789c656ea8e72ad13237bc"} Nov 03 19:42:14.362475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.362449 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:14.424252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.424209 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca\") pod \"c00fa23d-8cc1-49df-b163-569fb54e26c8\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " Nov 03 19:42:14.424441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.424311 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf\" (UniqueName: \"kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf\") pod \"c00fa23d-8cc1-49df-b163-569fb54e26c8\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " Nov 03 19:42:14.424441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.424393 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca\") pod \"c00fa23d-8cc1-49df-b163-569fb54e26c8\" (UID: \"c00fa23d-8cc1-49df-b163-569fb54e26c8\") " Nov 03 19:42:14.424674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.424625 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca" pod "c00fa23d-8cc1-49df-b163-569fb54e26c8" (UID: "c00fa23d-8cc1-49df-b163-569fb54e26c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:14.425142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.425118 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:14.426482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.426461 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf" pod "c00fa23d-8cc1-49df-b163-569fb54e26c8" (UID: "c00fa23d-8cc1-49df-b163-569fb54e26c8"). InnerVolumeSpecName "kube-api-access-2d9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:14.426832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.426809 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca" pod "c00fa23d-8cc1-49df-b163-569fb54e26c8" (UID: "c00fa23d-8cc1-49df-b163-569fb54e26c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:14.527041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.527013 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9gf\" (UniqueName: \"kubernetes.io/projected/c00fa23d-8cc1-49df-b163-569fb54e26c8-kube-api-access-2d9gf\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:14.527041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:14.527041 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00fa23d-8cc1-49df-b163-569fb54e26c8-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:15.187361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.187329 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" Nov 03 19:42:15.187572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.187411 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv" event={"ID":"c00fa23d-8cc1-49df-b163-569fb54e26c8","Type":"ContainerDied","Data":"a7b86baf0b1b8db6d45fbead98c2c729bb60aa82f4ad705f18670f4736866b0c"} Nov 03 19:42:15.187572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.187525 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b86baf0b1b8db6d45fbead98c2c729bb60aa82f4ad705f18670f4736866b0c" Nov 03 19:42:15.616231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.616192 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s"] Nov 03 19:42:15.617066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.617045 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c00fa23d-8cc1-49df-b163-569fb54e26c8" containerName="revoke-keys" Nov 03 19:42:15.617066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.617065 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00fa23d-8cc1-49df-b163-569fb54e26c8" containerName="revoke-keys" Nov 03 19:42:15.617252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.617238 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00fa23d-8cc1-49df-b163-569fb54e26c8" containerName="revoke-keys" Nov 03 19:42:15.622358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.622337 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.632734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.632188 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:42:15.633425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.633401 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:42:15.639137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.639114 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s"] Nov 03 19:42:15.660049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.659567 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.660049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.659615 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqq59\" (UniqueName: \"kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.660049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.659925 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.761097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.761067 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.761256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.761199 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.761256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.761235 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqq59\" (UniqueName: \"kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.762874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.762844 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.767455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.767406 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.789209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.789137 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqq59\" (UniqueName: \"kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59\") pod \"maas-api-revoke-sub-418d8691-v2r5s\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:15.940077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:15.939994 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:16.231578 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:16.231543 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda146c62a_01ca_4a08_963c_3d9636225950.slice/crio-b48393ea85e3c279f9d6c2aacc7734fb46046740ca08658dd5265840f413d806 WatchSource:0}: Error finding container b48393ea85e3c279f9d6c2aacc7734fb46046740ca08658dd5265840f413d806: Status 404 returned error can't find the container with id b48393ea85e3c279f9d6c2aacc7734fb46046740ca08658dd5265840f413d806 Nov 03 19:42:16.234141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:16.234113 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s"] Nov 03 19:42:17.207637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:17.207599 5233 generic.go:356] "Generic (PLEG): container finished" podID="a146c62a-01ca-4a08-963c-3d9636225950" containerID="fd86cabf65cefbcfe7f4e35d947e8a53844269da6d293a499bf836faba0314c7" exitCode=0 Nov 03 19:42:17.208092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:17.207676 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" event={"ID":"a146c62a-01ca-4a08-963c-3d9636225950","Type":"ContainerDied","Data":"fd86cabf65cefbcfe7f4e35d947e8a53844269da6d293a499bf836faba0314c7"} Nov 03 19:42:17.208092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:17.207758 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" event={"ID":"a146c62a-01ca-4a08-963c-3d9636225950","Type":"ContainerStarted","Data":"b48393ea85e3c279f9d6c2aacc7734fb46046740ca08658dd5265840f413d806"} Nov 03 19:42:18.223115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.223082 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj"] Nov 03 19:42:18.227729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.227706 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.237113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.237090 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj"] Nov 03 19:42:18.304810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.304783 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f5j\" (UniqueName: \"kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.304947 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.304861 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.305182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.305149 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.421729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.421692 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.422000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.421758 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f5j\" (UniqueName: \"kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.422000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.421837 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.430640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.430600 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.451354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.447589 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f5j\" (UniqueName: \"kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.459695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.459656 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2039ceec-gm2cj\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.488373 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.488348 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:18.556245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.556218 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:18.625023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.624955 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca\") pod \"a146c62a-01ca-4a08-963c-3d9636225950\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " Nov 03 19:42:18.625224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.625201 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59\" (UniqueName: \"kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59\") pod \"a146c62a-01ca-4a08-963c-3d9636225950\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " Nov 03 19:42:18.625330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.625312 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca\") pod \"a146c62a-01ca-4a08-963c-3d9636225950\" (UID: \"a146c62a-01ca-4a08-963c-3d9636225950\") " Nov 03 19:42:18.626391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.626331 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca" pod "a146c62a-01ca-4a08-963c-3d9636225950" (UID: "a146c62a-01ca-4a08-963c-3d9636225950"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:18.642450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.642420 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59" pod "a146c62a-01ca-4a08-963c-3d9636225950" (UID: "a146c62a-01ca-4a08-963c-3d9636225950"). InnerVolumeSpecName "kube-api-access-rqq59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:18.644451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.644422 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca" pod "a146c62a-01ca-4a08-963c-3d9636225950" (UID: "a146c62a-01ca-4a08-963c-3d9636225950"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:18.726877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.726794 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqq59\" (UniqueName: \"kubernetes.io/projected/a146c62a-01ca-4a08-963c-3d9636225950-kube-api-access-rqq59\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:18.726877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.726830 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:18.726877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.726841 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a146c62a-01ca-4a08-963c-3d9636225950-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:18.844002 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:18.843942 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb18892_f74e_4a1c_96e6_d39410a5686b.slice/crio-5ad91bf826466342a1af1a1a6b50ab3d2e9db72431f496364a578f9dd2f5ab5b WatchSource:0}: Error finding container 5ad91bf826466342a1af1a1a6b50ab3d2e9db72431f496364a578f9dd2f5ab5b: Status 404 returned error can't find the container with id 5ad91bf826466342a1af1a1a6b50ab3d2e9db72431f496364a578f9dd2f5ab5b Nov 03 19:42:18.844716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:18.844691 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj"] Nov 03 19:42:19.230262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:19.230234 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" Nov 03 19:42:19.230262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:19.230244 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s" event={"ID":"a146c62a-01ca-4a08-963c-3d9636225950","Type":"ContainerDied","Data":"b48393ea85e3c279f9d6c2aacc7734fb46046740ca08658dd5265840f413d806"} Nov 03 19:42:19.230843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:19.230335 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b48393ea85e3c279f9d6c2aacc7734fb46046740ca08658dd5265840f413d806" Nov 03 19:42:19.232647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:19.232621 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" event={"ID":"dfb18892-f74e-4a1c-96e6-d39410a5686b","Type":"ContainerStarted","Data":"5ad91bf826466342a1af1a1a6b50ab3d2e9db72431f496364a578f9dd2f5ab5b"} Nov 03 19:42:20.243579 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:20.243540 5233 generic.go:356] "Generic (PLEG): container finished" podID="dfb18892-f74e-4a1c-96e6-d39410a5686b" containerID="c603d79f6b12fa8e0e8ef089c2323a43463ddefd3df42274c8228ebd6721b646" exitCode=0 Nov 03 19:42:20.243935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:20.243615 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" event={"ID":"dfb18892-f74e-4a1c-96e6-d39410a5686b","Type":"ContainerDied","Data":"c603d79f6b12fa8e0e8ef089c2323a43463ddefd3df42274c8228ebd6721b646"} Nov 03 19:42:21.219262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.219229 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww"] Nov 03 19:42:21.219939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.219920 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a146c62a-01ca-4a08-963c-3d9636225950" containerName="revoke-keys" Nov 03 19:42:21.220070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.219940 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a146c62a-01ca-4a08-963c-3d9636225950" containerName="revoke-keys" Nov 03 19:42:21.220132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.220087 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a146c62a-01ca-4a08-963c-3d9636225950" containerName="revoke-keys" Nov 03 19:42:21.224143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.224118 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.234886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.233319 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww"] Nov 03 19:42:21.280358 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:42:21.280290 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:42:21.368695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.368662 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.368872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.368733 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.368966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.368913 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.430763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.430738 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:21.471698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.471602 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.471886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.471792 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.472275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.472245 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.474044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.474020 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.477135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.477101 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.515005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.506664 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv\") pod \"maas-api-revoke-sub-a5c2571a-h6qww\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.553469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.552731 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:21.573892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.573863 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j\" (UniqueName: \"kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j\") pod \"dfb18892-f74e-4a1c-96e6-d39410a5686b\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " Nov 03 19:42:21.574082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.573946 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca\") pod \"dfb18892-f74e-4a1c-96e6-d39410a5686b\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " Nov 03 19:42:21.574082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.573994 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca\") pod \"dfb18892-f74e-4a1c-96e6-d39410a5686b\" (UID: \"dfb18892-f74e-4a1c-96e6-d39410a5686b\") " Nov 03 19:42:21.574668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.574641 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca" pod "dfb18892-f74e-4a1c-96e6-d39410a5686b" (UID: "dfb18892-f74e-4a1c-96e6-d39410a5686b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:21.576443 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.576417 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca" pod "dfb18892-f74e-4a1c-96e6-d39410a5686b" (UID: "dfb18892-f74e-4a1c-96e6-d39410a5686b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:21.577031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.577003 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j" pod "dfb18892-f74e-4a1c-96e6-d39410a5686b" (UID: "dfb18892-f74e-4a1c-96e6-d39410a5686b"). InnerVolumeSpecName "kube-api-access-k4f5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:21.677421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.677384 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:21.677421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.677421 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4f5j\" (UniqueName: \"kubernetes.io/projected/dfb18892-f74e-4a1c-96e6-d39410a5686b-kube-api-access-k4f5j\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:21.677647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.677437 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb18892-f74e-4a1c-96e6-d39410a5686b-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:21.857781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:21.857755 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww"] Nov 03 19:42:21.858457 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:21.858431 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc78a744_e33c_4fd2_89d1_986f015fa4b6.slice/crio-787106ed2b46e10888882a005484180b364931192c9d231fd4decf699c50092b WatchSource:0}: Error finding container 787106ed2b46e10888882a005484180b364931192c9d231fd4decf699c50092b: Status 404 returned error can't find the container with id 787106ed2b46e10888882a005484180b364931192c9d231fd4decf699c50092b Nov 03 19:42:22.267155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:22.267112 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" event={"ID":"cc78a744-e33c-4fd2-89d1-986f015fa4b6","Type":"ContainerStarted","Data":"787106ed2b46e10888882a005484180b364931192c9d231fd4decf699c50092b"} Nov 03 19:42:22.268753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:22.268728 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" Nov 03 19:42:22.268870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:22.268758 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj" event={"ID":"dfb18892-f74e-4a1c-96e6-d39410a5686b","Type":"ContainerDied","Data":"5ad91bf826466342a1af1a1a6b50ab3d2e9db72431f496364a578f9dd2f5ab5b"} Nov 03 19:42:22.268870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:22.268797 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad91bf826466342a1af1a1a6b50ab3d2e9db72431f496364a578f9dd2f5ab5b" Nov 03 19:42:23.279668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:23.279636 5233 generic.go:356] "Generic (PLEG): container finished" podID="cc78a744-e33c-4fd2-89d1-986f015fa4b6" containerID="fa53696fb44e29ac174be16207b10eefe105105833183e0d21ef6cb7aa567674" exitCode=0 Nov 03 19:42:23.280086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:23.279729 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" event={"ID":"cc78a744-e33c-4fd2-89d1-986f015fa4b6","Type":"ContainerDied","Data":"fa53696fb44e29ac174be16207b10eefe105105833183e0d21ef6cb7aa567674"} Nov 03 19:42:24.480078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.480052 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:24.638170 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.638082 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv\") pod \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " Nov 03 19:42:24.638327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.638199 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca\") pod \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " Nov 03 19:42:24.638406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.638390 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca\") pod \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\" (UID: \"cc78a744-e33c-4fd2-89d1-986f015fa4b6\") " Nov 03 19:42:24.638638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.638610 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca" pod "cc78a744-e33c-4fd2-89d1-986f015fa4b6" (UID: "cc78a744-e33c-4fd2-89d1-986f015fa4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:24.639419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.639396 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:24.640352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.640328 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv" pod "cc78a744-e33c-4fd2-89d1-986f015fa4b6" (UID: "cc78a744-e33c-4fd2-89d1-986f015fa4b6"). InnerVolumeSpecName "kube-api-access-h8nkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:24.640481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.640464 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca" pod "cc78a744-e33c-4fd2-89d1-986f015fa4b6" (UID: "cc78a744-e33c-4fd2-89d1-986f015fa4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:24.674258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.674222 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx"] Nov 03 19:42:24.675011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.674960 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cc78a744-e33c-4fd2-89d1-986f015fa4b6" containerName="revoke-keys" Nov 03 19:42:24.675011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.674996 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc78a744-e33c-4fd2-89d1-986f015fa4b6" containerName="revoke-keys" Nov 03 19:42:24.675176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.675017 5233 cpu_manager.go:414] "containerMap: removing container" podUID="dfb18892-f74e-4a1c-96e6-d39410a5686b" containerName="revoke-keys" Nov 03 19:42:24.675176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.675025 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb18892-f74e-4a1c-96e6-d39410a5686b" containerName="revoke-keys" Nov 03 19:42:24.675278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.675198 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc78a744-e33c-4fd2-89d1-986f015fa4b6" containerName="revoke-keys" Nov 03 19:42:24.675278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.675212 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb18892-f74e-4a1c-96e6-d39410a5686b" containerName="revoke-keys" Nov 03 19:42:24.690331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.690307 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.693691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.693556 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx"] Nov 03 19:42:24.740368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.740334 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc78a744-e33c-4fd2-89d1-986f015fa4b6-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:24.740368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.740363 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/cc78a744-e33c-4fd2-89d1-986f015fa4b6-kube-api-access-h8nkv\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:24.841325 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841292 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2t6\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841396 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841427 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841616 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841513 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841616 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841532 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841627 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841664 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841701 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.841874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.841771 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.942918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.942812 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.942918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.942875 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.942961 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.943115 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2t6\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.943212 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.943254 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.943364 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.943390 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.943667 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.943485 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.944385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.944359 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.944935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.944907 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.945120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.944928 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.945245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.945223 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.950553 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.948274 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.950553 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.948358 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.958380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.958351 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.968940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.968907 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:24.969192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:24.969165 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2t6\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6\") pod \"e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:25.022379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.022347 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:25.310910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.310874 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" event={"ID":"cc78a744-e33c-4fd2-89d1-986f015fa4b6","Type":"ContainerDied","Data":"787106ed2b46e10888882a005484180b364931192c9d231fd4decf699c50092b"} Nov 03 19:42:25.311243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.310935 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww" Nov 03 19:42:25.311243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.310982 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787106ed2b46e10888882a005484180b364931192c9d231fd4decf699c50092b" Nov 03 19:42:25.329365 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:25.329333 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baec845_1de4_466c_82e8_2920589ba88d.slice/crio-feb5d771537cb4a4573d58972373821dd14f506b9f66b19c41220d16e87dca18 WatchSource:0}: Error finding container feb5d771537cb4a4573d58972373821dd14f506b9f66b19c41220d16e87dca18: Status 404 returned error can't find the container with id feb5d771537cb4a4573d58972373821dd14f506b9f66b19c41220d16e87dca18 Nov 03 19:42:25.332186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.332149 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:25.332300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.332218 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:25.332300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.332254 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:25.333804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.333776 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx"] Nov 03 19:42:25.392850 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.392820 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn"] Nov 03 19:42:25.399624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.399602 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.423031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.421907 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn"] Nov 03 19:42:25.553358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553324 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553395 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553519 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553553 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553643 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553709 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553803 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553852 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtmt\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.553985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.553872 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.660661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.660110 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.660661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.660200 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.660661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.660276 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.660661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.660309 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.660661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.660363 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.679146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.679118 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.679383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.679366 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.679548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.679532 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtmt\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.679676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.679658 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.679842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.679820 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.680519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.680495 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.680867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.680848 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.687846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.681254 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.692616 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.682811 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.692616 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.689483 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.694333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.694307 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtmt\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.695527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.695487 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.704703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.704677 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy\") pod \"e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:25.728046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:25.728020 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:26.096599 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:26.096570 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f360ec9_3afe_4707_96f3_b2b2f134bca7.slice/crio-4e05ab5c45109cf3353f62fdfd6536ae596b6097490d4bb1f29b1eddf30b973f WatchSource:0}: Error finding container 4e05ab5c45109cf3353f62fdfd6536ae596b6097490d4bb1f29b1eddf30b973f: Status 404 returned error can't find the container with id 4e05ab5c45109cf3353f62fdfd6536ae596b6097490d4bb1f29b1eddf30b973f Nov 03 19:42:26.099482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.099440 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:26.099570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.099534 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:26.099606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.099575 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:26.100313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.100289 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn"] Nov 03 19:42:26.341010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.340259 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" event={"ID":"2baec845-1de4-466c-82e8-2920589ba88d","Type":"ContainerStarted","Data":"8fdc7329c54f76d09bf5f93e5dd82f38ac8321f8412677b4f6570b2c28f75fa4"} Nov 03 19:42:26.341010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.340308 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" event={"ID":"2baec845-1de4-466c-82e8-2920589ba88d","Type":"ContainerStarted","Data":"feb5d771537cb4a4573d58972373821dd14f506b9f66b19c41220d16e87dca18"} Nov 03 19:42:26.343008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.342898 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" event={"ID":"8f360ec9-3afe-4707-96f3-b2b2f134bca7","Type":"ContainerStarted","Data":"b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6"} Nov 03 19:42:26.343201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.343184 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" event={"ID":"8f360ec9-3afe-4707-96f3-b2b2f134bca7","Type":"ContainerStarted","Data":"4e05ab5c45109cf3353f62fdfd6536ae596b6097490d4bb1f29b1eddf30b973f"} Nov 03 19:42:26.368209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.367506 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podStartSLOduration=2.367489777 podStartE2EDuration="2.367489777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:26.364435081 +0000 UTC m=+912.705548567" watchObservedRunningTime="2026-11-03 19:42:26.367489777 +0000 UTC m=+912.708603263" Nov 03 19:42:26.733993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:26.731191 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:27.025065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:27.024233 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:27.025481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:27.025460 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:27.025640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:27.025618 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:27.751257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:27.751217 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:27.753133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:27.753092 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:28.033480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:28.033390 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:28.033754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:28.033721 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:28.732809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:28.732773 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:28.733115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:28.733084 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:29.026953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:29.026069 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:29.026953 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:29.026131 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:29.729467 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:29.729429 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:29.729879 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:29.729837 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:30.052288 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.052194 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:30.052876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.052838 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:30.316868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.316766 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:42:30.317510 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.317471 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:30.370420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.370377 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:30.427140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.427081 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:30.451266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.451210 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:30.671053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.670523 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:30.742993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.742463 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:30.742993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.742537 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:30.821995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.821219 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:30.993886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:30.993124 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:42:31.025810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.025367 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:31.025810 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.025430 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:31.325162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.325060 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:42:31.346590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.326481 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:42:31.413998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.413203 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podStartSLOduration=6.413181746 podStartE2EDuration="6.413181746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:26.403329038 +0000 UTC m=+912.744442593" watchObservedRunningTime="2026-11-03 19:42:31.413181746 +0000 UTC m=+917.754295232" Nov 03 19:42:31.487001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.445194 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:42:31.527041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.487626 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:42:31.527041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.477849 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:42:31.527041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.479429 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded" Nov 03 19:42:31.527041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.489524 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:42:31.693074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.688959 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:31.706015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.705289 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:42:31.778768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.778720 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:31.780831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.780801 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:31.872031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.871986 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:31.872670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.824422 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:42:31.881250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:31.824133 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:42:32.072592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.072555 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:32.082206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.082170 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:32.082471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.082442 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:32.361059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.328321 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:32.508930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.508224 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:32.513048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.513009 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:42:32.803762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.803702 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:32.813869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.813827 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:42:32.827258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.826923 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:42:33.028004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.819108 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:33.036996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.883157 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:33.044241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.041300 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:42:33.044241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.041393 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:42:33.044241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.883293 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:33.044241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:32.883414 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 19:42:33.085110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.082901 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:33.085110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.082958 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:33.148136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.126933 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:33.148136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.129035 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:33.148136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.129094 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:33.385066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.383238 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:33.546000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.515451 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:33.546542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.519187 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:42:33.777258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.777214 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:33.786091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.786047 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:33.835991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:33.833905 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:42:34.033622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.033523 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:34.045221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.045184 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:34.045698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.045677 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:42:34.110101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.110067 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:34.110342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.110318 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:34.112066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.111955 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:42:34.124077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.124045 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:42:34.173777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.168173 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:34.345090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.342322 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:42:34.364025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.359809 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:42:34.480462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.480429 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:42:34.481071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.481054 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:42:34.548324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.529107 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:42:34.549119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.549071 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:42:34.732115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.731902 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:34.732394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.732363 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:34.760354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:34.759388 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:42:35.048672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.046776 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:42:35.048672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.046847 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:42:35.744092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.734445 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:35.744092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.734502 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:35.745839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.745809 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx"] Nov 03 19:42:35.842117 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.842085 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:35.886990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.869140 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:42:35.886990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.872605 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:42:35.886990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.886314 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx"] Nov 03 19:42:35.965146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.965112 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsg9\" (UniqueName: \"kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:35.965360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.965291 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:35.965360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:35.965345 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.066650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.066562 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.066836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.066655 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.066836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.066775 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsg9\" (UniqueName: \"kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.067873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.067845 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.070555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.070534 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.083400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.083372 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsg9\" (UniqueName: \"kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9\") pod \"maas-api-revoke-sub-2cf76014-8r5wx\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.248379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.248345 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:36.669510 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.668240 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx"] Nov 03 19:42:36.753782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.753322 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:42:36.753782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.753383 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:42:36.799335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.799300 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" event={"ID":"bb2a2460-49bd-4bd2-a935-d3fd58f497ce","Type":"ContainerStarted","Data":"53d00072e6497d80d19e046ec23a55c4528823e39bf81e8d6e424c22551cdcce"} Nov 03 19:42:36.823197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.818296 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:36.823197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.819839 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:36.827997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:36.827921 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:42:37.023191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.023162 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds"] Nov 03 19:42:37.047180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.047150 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds"] Nov 03 19:42:37.051000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.047457 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.165335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.165299 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlt2\" (UniqueName: \"kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.165564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.165540 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.165636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.165592 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.267028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.266931 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.267028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.267010 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.267224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.267211 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlt2\" (UniqueName: \"kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.267902 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.267874 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.269437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.269412 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.283240 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.283215 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlt2\" (UniqueName: \"kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2\") pod \"maas-api-revoke-sub-6fa7a1fe-vswds\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.434165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.433422 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:37.608234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.608197 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8"] Nov 03 19:42:37.614349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.614323 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.633239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.633210 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8"] Nov 03 19:42:37.676007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.675941 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.676205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.676185 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn77x\" (UniqueName: \"kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.676324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.676302 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.676888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.676869 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr"] Nov 03 19:42:37.696044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.681905 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.708263 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.708236 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr"] Nov 03 19:42:37.734097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.734070 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:37.735278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.735256 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:37.736187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.736172 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:42:37.777136 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:37.777106 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47af1e84_6b91_4dbb_8a09_4880b8dd2825.slice/crio-5bbec0483494783af10c7452060f153420f4aa9c89d2ca32d4828293300b1b59 WatchSource:0}: Error finding container 5bbec0483494783af10c7452060f153420f4aa9c89d2ca32d4828293300b1b59: Status 404 returned error can't find the container with id 5bbec0483494783af10c7452060f153420f4aa9c89d2ca32d4828293300b1b59 Nov 03 19:42:37.777568 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.777544 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.777615 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.777596 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.777782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.777760 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds"] Nov 03 19:42:37.777845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.777792 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.777936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.777913 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn77x\" (UniqueName: \"kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.778073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.778027 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266hn\" (UniqueName: \"kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.778073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.778063 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.778339 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.778321 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.780540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.780512 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.797060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.791294 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn77x\" (UniqueName: \"kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x\") pod \"maas-api-revoke-sub-07a9c610-469z8\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:37.815930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.815562 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" event={"ID":"bb2a2460-49bd-4bd2-a935-d3fd58f497ce","Type":"ContainerStarted","Data":"31ea3755970852343b23961de1a89ce18df3c28e3b6a3b3fcaab77cc82853d6f"} Nov 03 19:42:37.820288 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.819511 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" event={"ID":"47af1e84-6b91-4dbb-8a09-4880b8dd2825","Type":"ContainerStarted","Data":"5bbec0483494783af10c7452060f153420f4aa9c89d2ca32d4828293300b1b59"} Nov 03 19:42:37.842197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.841521 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" podStartSLOduration=2.84150279 podStartE2EDuration="2.84150279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:37.837678045 +0000 UTC m=+924.178791535" watchObservedRunningTime="2026-11-03 19:42:37.84150279 +0000 UTC m=+924.182616276" Nov 03 19:42:37.879108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.879076 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.879344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.879323 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266hn\" (UniqueName: \"kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.879528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.879507 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.880322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.880298 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.881881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.881859 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.892300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.892277 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266hn\" (UniqueName: \"kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn\") pod \"maas-api-revoke-sub-a0c646bf-zntvr\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:37.946738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:37.939693 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:38.021050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.020774 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:38.251119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.251087 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8"] Nov 03 19:42:38.339519 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:38.338248 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe158e4_2340_42e2_a828_f6e0fb869225.slice/crio-e857bcdc9a73d3f9fb93b7d7c8c8e8545d3dc8a29ff39a5dd6eaba5d0ffd05c7 WatchSource:0}: Error finding container e857bcdc9a73d3f9fb93b7d7c8c8e8545d3dc8a29ff39a5dd6eaba5d0ffd05c7: Status 404 returned error can't find the container with id e857bcdc9a73d3f9fb93b7d7c8c8e8545d3dc8a29ff39a5dd6eaba5d0ffd05c7 Nov 03 19:42:38.340429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.340400 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr"] Nov 03 19:42:38.829718 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.829681 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" event={"ID":"dbe158e4-2340-42e2-a828-f6e0fb869225","Type":"ContainerStarted","Data":"524c17871f3818e20b52fa7e58f70a2b23e3250dabf90b1710a5cce81afc5601"} Nov 03 19:42:38.830252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.829728 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" event={"ID":"dbe158e4-2340-42e2-a828-f6e0fb869225","Type":"ContainerStarted","Data":"e857bcdc9a73d3f9fb93b7d7c8c8e8545d3dc8a29ff39a5dd6eaba5d0ffd05c7"} Nov 03 19:42:38.831773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.831744 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" event={"ID":"8d9a689c-bdb4-449a-ad06-39af368f8385","Type":"ContainerStarted","Data":"50ce50c87507f42b515f90dd98e17dbedd1f0866ba46f19826080a152e836435"} Nov 03 19:42:38.831907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.831802 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" event={"ID":"8d9a689c-bdb4-449a-ad06-39af368f8385","Type":"ContainerStarted","Data":"00bc7796692ebecc9d9f617a29c125713788c368e22bff9f265583ef07a534a7"} Nov 03 19:42:38.834040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.833753 5233 generic.go:356] "Generic (PLEG): container finished" podID="bb2a2460-49bd-4bd2-a935-d3fd58f497ce" containerID="31ea3755970852343b23961de1a89ce18df3c28e3b6a3b3fcaab77cc82853d6f" exitCode=0 Nov 03 19:42:38.834040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.833918 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" event={"ID":"bb2a2460-49bd-4bd2-a935-d3fd58f497ce","Type":"ContainerDied","Data":"31ea3755970852343b23961de1a89ce18df3c28e3b6a3b3fcaab77cc82853d6f"} Nov 03 19:42:38.836114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.836084 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" event={"ID":"47af1e84-6b91-4dbb-8a09-4880b8dd2825","Type":"ContainerStarted","Data":"3cba72afa1ed947df6677931993958fc280098f7cf39df5fa5c454267787d461"} Nov 03 19:42:38.851197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.851157 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" podStartSLOduration=1.851144548 podStartE2EDuration="1.851144548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:38.848681476 +0000 UTC m=+925.189794961" watchObservedRunningTime="2026-11-03 19:42:38.851144548 +0000 UTC m=+925.192258034" Nov 03 19:42:38.872692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.872653 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" podStartSLOduration=2.87263926 podStartE2EDuration="2.87263926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:38.866662343 +0000 UTC m=+925.207775830" watchObservedRunningTime="2026-11-03 19:42:38.87263926 +0000 UTC m=+925.213752790" Nov 03 19:42:38.913914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:38.913859 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" podStartSLOduration=1.91384358 podStartE2EDuration="1.91384358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:38.907912567 +0000 UTC m=+925.249026076" watchObservedRunningTime="2026-11-03 19:42:38.91384358 +0000 UTC m=+925.254957067" Nov 03 19:42:39.868150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.868054 5233 generic.go:356] "Generic (PLEG): container finished" podID="47af1e84-6b91-4dbb-8a09-4880b8dd2825" containerID="3cba72afa1ed947df6677931993958fc280098f7cf39df5fa5c454267787d461" exitCode=0 Nov 03 19:42:39.868150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.868093 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" event={"ID":"47af1e84-6b91-4dbb-8a09-4880b8dd2825","Type":"ContainerDied","Data":"3cba72afa1ed947df6677931993958fc280098f7cf39df5fa5c454267787d461"} Nov 03 19:42:39.870443 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.870416 5233 generic.go:356] "Generic (PLEG): container finished" podID="dbe158e4-2340-42e2-a828-f6e0fb869225" containerID="524c17871f3818e20b52fa7e58f70a2b23e3250dabf90b1710a5cce81afc5601" exitCode=0 Nov 03 19:42:39.870588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.870500 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" event={"ID":"dbe158e4-2340-42e2-a828-f6e0fb869225","Type":"ContainerDied","Data":"524c17871f3818e20b52fa7e58f70a2b23e3250dabf90b1710a5cce81afc5601"} Nov 03 19:42:39.872399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.872375 5233 generic.go:356] "Generic (PLEG): container finished" podID="8d9a689c-bdb4-449a-ad06-39af368f8385" containerID="50ce50c87507f42b515f90dd98e17dbedd1f0866ba46f19826080a152e836435" exitCode=0 Nov 03 19:42:39.872487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.872453 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" event={"ID":"8d9a689c-bdb4-449a-ad06-39af368f8385","Type":"ContainerDied","Data":"50ce50c87507f42b515f90dd98e17dbedd1f0866ba46f19826080a152e836435"} Nov 03 19:42:39.913995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.913944 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2"] Nov 03 19:42:39.920177 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.920141 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:39.954800 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:39.952810 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2"] Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021259 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021442 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6f7\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021552 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021582 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021629 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021675 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021704 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021748 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.022110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.021836 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.075215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.075183 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:40.124926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.124843 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca\") pod \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " Nov 03 19:42:40.125106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.124945 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca\") pod \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " Nov 03 19:42:40.125180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125139 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9\" (UniqueName: \"kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9\") pod \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\" (UID: \"bb2a2460-49bd-4bd2-a935-d3fd58f497ce\") " Nov 03 19:42:40.125289 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125266 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca" pod "bb2a2460-49bd-4bd2-a935-d3fd58f497ce" (UID: "bb2a2460-49bd-4bd2-a935-d3fd58f497ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:40.125403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125383 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.125516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125422 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.125516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125500 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.125614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125599 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.125672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125648 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.125842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125823 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6f7\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.125987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125952 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.126062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.126005 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.126062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.126055 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.126165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.126148 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:40.127077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.125823 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.127077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.127030 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.127247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.127112 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.127369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.127321 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.127685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.127631 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.129803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.129759 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.130373 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.130349 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.131344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.131317 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca" pod "bb2a2460-49bd-4bd2-a935-d3fd58f497ce" (UID: "bb2a2460-49bd-4bd2-a935-d3fd58f497ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:40.136044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.132172 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9" pod "bb2a2460-49bd-4bd2-a935-d3fd58f497ce" (UID: "bb2a2460-49bd-4bd2-a935-d3fd58f497ce"). InnerVolumeSpecName "kube-api-access-nhsg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:40.141520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.141494 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6f7\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.141909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.141893 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.228007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.227951 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsg9\" (UniqueName: \"kubernetes.io/projected/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-kube-api-access-nhsg9\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:40.228007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.228005 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2a2460-49bd-4bd2-a935-d3fd58f497ce-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:40.243268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.243240 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:40.546631 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:40.546591 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde74f23f_0e40_4ae6_be24_e052f803e237.slice/crio-93b8ac97c53eae01133783c8aba5e4e4fe3caba7a0d2e3cf1b87b7dc5da63bf8 WatchSource:0}: Error finding container 93b8ac97c53eae01133783c8aba5e4e4fe3caba7a0d2e3cf1b87b7dc5da63bf8: Status 404 returned error can't find the container with id 93b8ac97c53eae01133783c8aba5e4e4fe3caba7a0d2e3cf1b87b7dc5da63bf8 Nov 03 19:42:40.547324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.547255 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2"] Nov 03 19:42:40.549365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.549330 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:40.549461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.549396 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:40.549461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.549425 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:40.886025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.885871 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" event={"ID":"de74f23f-0e40-4ae6-be24-e052f803e237","Type":"ContainerStarted","Data":"044b140e395aa8c4bd5f16d25d22bf72eea13d33149bdb3a1170ecc3860ccc50"} Nov 03 19:42:40.886025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.885944 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" event={"ID":"de74f23f-0e40-4ae6-be24-e052f803e237","Type":"ContainerStarted","Data":"93b8ac97c53eae01133783c8aba5e4e4fe3caba7a0d2e3cf1b87b7dc5da63bf8"} Nov 03 19:42:40.887761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.887731 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" Nov 03 19:42:40.887879 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.887770 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx" event={"ID":"bb2a2460-49bd-4bd2-a935-d3fd58f497ce","Type":"ContainerDied","Data":"53d00072e6497d80d19e046ec23a55c4528823e39bf81e8d6e424c22551cdcce"} Nov 03 19:42:40.887879 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.887811 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d00072e6497d80d19e046ec23a55c4528823e39bf81e8d6e424c22551cdcce" Nov 03 19:42:40.932290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:40.932225 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podStartSLOduration=1.932202825 podStartE2EDuration="1.932202825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:40.910687218 +0000 UTC m=+927.251800703" watchObservedRunningTime="2026-11-03 19:42:40.932202825 +0000 UTC m=+927.273316311" Nov 03 19:42:41.107822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.107773 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.246608 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.246721 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.246764 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.248585 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn\" (UniqueName: \"kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn\") pod \"dbe158e4-2340-42e2-a828-f6e0fb869225\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.248805 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca\") pod \"dbe158e4-2340-42e2-a828-f6e0fb869225\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.248957 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca\") pod \"dbe158e4-2340-42e2-a828-f6e0fb869225\" (UID: \"dbe158e4-2340-42e2-a828-f6e0fb869225\") " Nov 03 19:42:41.257791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.255570 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca" pod "dbe158e4-2340-42e2-a828-f6e0fb869225" (UID: "dbe158e4-2340-42e2-a828-f6e0fb869225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:41.260172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.260145 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn" pod "dbe158e4-2340-42e2-a828-f6e0fb869225" (UID: "dbe158e4-2340-42e2-a828-f6e0fb869225"). InnerVolumeSpecName "kube-api-access-266hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:41.264150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.264122 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca" pod "dbe158e4-2340-42e2-a828-f6e0fb869225" (UID: "dbe158e4-2340-42e2-a828-f6e0fb869225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:41.322491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.322461 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:41.325668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.325646 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:41.351688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.351647 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.351688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.351685 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266hn\" (UniqueName: \"kubernetes.io/projected/dbe158e4-2340-42e2-a828-f6e0fb869225-kube-api-access-266hn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.351920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.351701 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe158e4-2340-42e2-a828-f6e0fb869225-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.453029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.452946 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca\") pod \"8d9a689c-bdb4-449a-ad06-39af368f8385\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " Nov 03 19:42:41.453168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453125 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca\") pod \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " Nov 03 19:42:41.453272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453254 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x\" (UniqueName: \"kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x\") pod \"8d9a689c-bdb4-449a-ad06-39af368f8385\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " Nov 03 19:42:41.453336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453313 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca" pod "8d9a689c-bdb4-449a-ad06-39af368f8385" (UID: "8d9a689c-bdb4-449a-ad06-39af368f8385"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:41.453381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453358 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca\") pod \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " Nov 03 19:42:41.453480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453460 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2\" (UniqueName: \"kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2\") pod \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\" (UID: \"47af1e84-6b91-4dbb-8a09-4880b8dd2825\") " Nov 03 19:42:41.453592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453573 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca\") pod \"8d9a689c-bdb4-449a-ad06-39af368f8385\" (UID: \"8d9a689c-bdb4-449a-ad06-39af368f8385\") " Nov 03 19:42:41.453695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.453670 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca" pod "47af1e84-6b91-4dbb-8a09-4880b8dd2825" (UID: "47af1e84-6b91-4dbb-8a09-4880b8dd2825"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:41.454201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.454175 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.454312 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.454203 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.455665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.455639 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x" pod "8d9a689c-bdb4-449a-ad06-39af368f8385" (UID: "8d9a689c-bdb4-449a-ad06-39af368f8385"). InnerVolumeSpecName "kube-api-access-wn77x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:41.455936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.455912 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca" pod "47af1e84-6b91-4dbb-8a09-4880b8dd2825" (UID: "47af1e84-6b91-4dbb-8a09-4880b8dd2825"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:41.455936 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.455929 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2" pod "47af1e84-6b91-4dbb-8a09-4880b8dd2825" (UID: "47af1e84-6b91-4dbb-8a09-4880b8dd2825"). InnerVolumeSpecName "kube-api-access-qjlt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:41.456114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.456030 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca" pod "8d9a689c-bdb4-449a-ad06-39af368f8385" (UID: "8d9a689c-bdb4-449a-ad06-39af368f8385"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:41.555697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.555667 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlt2\" (UniqueName: \"kubernetes.io/projected/47af1e84-6b91-4dbb-8a09-4880b8dd2825-kube-api-access-qjlt2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.555697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.555695 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9a689c-bdb4-449a-ad06-39af368f8385-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.555697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.555706 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47af1e84-6b91-4dbb-8a09-4880b8dd2825-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.555932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.555715 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn77x\" (UniqueName: \"kubernetes.io/projected/8d9a689c-bdb4-449a-ad06-39af368f8385-kube-api-access-wn77x\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:41.773508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.773473 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8"] Nov 03 19:42:41.774417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774385 5233 cpu_manager.go:414] "containerMap: removing container" podUID="dbe158e4-2340-42e2-a828-f6e0fb869225" containerName="revoke-keys" Nov 03 19:42:41.774545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774534 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe158e4-2340-42e2-a828-f6e0fb869225" containerName="revoke-keys" Nov 03 19:42:41.774617 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774611 5233 cpu_manager.go:414] "containerMap: removing container" podUID="bb2a2460-49bd-4bd2-a935-d3fd58f497ce" containerName="revoke-keys" Nov 03 19:42:41.774670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774663 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2a2460-49bd-4bd2-a935-d3fd58f497ce" containerName="revoke-keys" Nov 03 19:42:41.774736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774730 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8d9a689c-bdb4-449a-ad06-39af368f8385" containerName="revoke-keys" Nov 03 19:42:41.774778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774772 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9a689c-bdb4-449a-ad06-39af368f8385" containerName="revoke-keys" Nov 03 19:42:41.774849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774842 5233 cpu_manager.go:414] "containerMap: removing container" podUID="47af1e84-6b91-4dbb-8a09-4880b8dd2825" containerName="revoke-keys" Nov 03 19:42:41.774901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.774894 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47af1e84-6b91-4dbb-8a09-4880b8dd2825" containerName="revoke-keys" Nov 03 19:42:41.775121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.775107 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9a689c-bdb4-449a-ad06-39af368f8385" containerName="revoke-keys" Nov 03 19:42:41.775238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.775226 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="47af1e84-6b91-4dbb-8a09-4880b8dd2825" containerName="revoke-keys" Nov 03 19:42:41.775322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.775312 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe158e4-2340-42e2-a828-f6e0fb869225" containerName="revoke-keys" Nov 03 19:42:41.775415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.775406 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2a2460-49bd-4bd2-a935-d3fd58f497ce" containerName="revoke-keys" Nov 03 19:42:41.781047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.781022 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.784307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.784278 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8"] Nov 03 19:42:41.872750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.872720 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.872869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.872844 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.873131 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.873102 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxs88\" (UniqueName: \"kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.918091 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:42:41.917842 5233 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9a689c_bdb4_449a_ad06_39af368f8385.slice/crio-00bc7796692ebecc9d9f617a29c125713788c368e22bff9f265583ef07a534a7\": RecentStats: unable to find data in memory cache]" Nov 03 19:42:41.930216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.930185 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" Nov 03 19:42:41.930393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.930190 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8" event={"ID":"8d9a689c-bdb4-449a-ad06-39af368f8385","Type":"ContainerDied","Data":"00bc7796692ebecc9d9f617a29c125713788c368e22bff9f265583ef07a534a7"} Nov 03 19:42:41.930393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.930327 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00bc7796692ebecc9d9f617a29c125713788c368e22bff9f265583ef07a534a7" Nov 03 19:42:41.931984 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.931922 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" Nov 03 19:42:41.932120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.931966 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds" event={"ID":"47af1e84-6b91-4dbb-8a09-4880b8dd2825","Type":"ContainerDied","Data":"5bbec0483494783af10c7452060f153420f4aa9c89d2ca32d4828293300b1b59"} Nov 03 19:42:41.932120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.932028 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bbec0483494783af10c7452060f153420f4aa9c89d2ca32d4828293300b1b59" Nov 03 19:42:41.934035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.934014 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" Nov 03 19:42:41.934129 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.934009 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr" event={"ID":"dbe158e4-2340-42e2-a828-f6e0fb869225","Type":"ContainerDied","Data":"e857bcdc9a73d3f9fb93b7d7c8c8e8545d3dc8a29ff39a5dd6eaba5d0ffd05c7"} Nov 03 19:42:41.934129 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.934074 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e857bcdc9a73d3f9fb93b7d7c8c8e8545d3dc8a29ff39a5dd6eaba5d0ffd05c7" Nov 03 19:42:41.974413 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.974382 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.974558 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.974481 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.974734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.974714 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxs88\" (UniqueName: \"kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.975285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.975259 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.979310 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.978083 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:41.982677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.982651 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2"] Nov 03 19:42:42.008995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:41.996389 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.016784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.016751 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxs88\" (UniqueName: \"kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88\") pod \"maas-api-revoke-sub-37cfa5ee-7zvc8\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:42.019475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.019362 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2"] Nov 03 19:42:42.076784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.076704 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.076928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.076821 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9r6f\" (UniqueName: \"kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.076928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.076879 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.158893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.158856 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:42.179143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.179108 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.179264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.179250 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9r6f\" (UniqueName: \"kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.179338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.179323 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.179819 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.179791 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.181751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.181720 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.215998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.210791 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9r6f\" (UniqueName: \"kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f\") pod \"maas-api-revoke-sub-2180aeb5-6f9b2\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.244364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.244334 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:42.244512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.244379 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:42.349861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.344401 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:42.525611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.522581 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8"] Nov 03 19:42:42.774608 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:42.774578 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147865af_a377_4379_8f42_c4d1d622ce91.slice/crio-aaa00a27c6a001230cc6ddc45893480066123f1f4c7ccad3c0196d6fc20867ea WatchSource:0}: Error finding container aaa00a27c6a001230cc6ddc45893480066123f1f4c7ccad3c0196d6fc20867ea: Status 404 returned error can't find the container with id aaa00a27c6a001230cc6ddc45893480066123f1f4c7ccad3c0196d6fc20867ea Nov 03 19:42:42.775415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.775391 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2"] Nov 03 19:42:42.944281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.944245 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" event={"ID":"eb30f743-6e82-4fa6-be71-7c79ee23aa1e","Type":"ContainerStarted","Data":"c1ea57cae3f19fbc7734e2f8d20c593c05be839190a29de51334f0cf072236fd"} Nov 03 19:42:42.945558 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:42.945530 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" event={"ID":"147865af-a377-4379-8f42-c4d1d622ce91","Type":"ContainerStarted","Data":"aaa00a27c6a001230cc6ddc45893480066123f1f4c7ccad3c0196d6fc20867ea"} Nov 03 19:42:43.244366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:43.244334 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:43.244552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:43.244397 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:43.974076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:43.974026 5233 generic.go:356] "Generic (PLEG): container finished" podID="eb30f743-6e82-4fa6-be71-7c79ee23aa1e" containerID="dc0e861b6d6ff5919ccad77f59f8a778cbbe148da9cd5fe4941b520a4422b0de" exitCode=0 Nov 03 19:42:43.974540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:43.974181 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" event={"ID":"eb30f743-6e82-4fa6-be71-7c79ee23aa1e","Type":"ContainerDied","Data":"dc0e861b6d6ff5919ccad77f59f8a778cbbe148da9cd5fe4941b520a4422b0de"} Nov 03 19:42:43.977101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:43.977069 5233 generic.go:356] "Generic (PLEG): container finished" podID="147865af-a377-4379-8f42-c4d1d622ce91" containerID="e4f1cf880974fe159a644e2c78b9ea12f257a354a49cc46f3f9e8b0a18b757bf" exitCode=0 Nov 03 19:42:43.977436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:43.977413 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" event={"ID":"147865af-a377-4379-8f42-c4d1d622ce91","Type":"ContainerDied","Data":"e4f1cf880974fe159a644e2c78b9ea12f257a354a49cc46f3f9e8b0a18b757bf"} Nov 03 19:42:44.244224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:44.244188 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:44.244400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:44.244247 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:45.243906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.243618 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:45.243906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.243686 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:45.330001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.325853 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:45.371495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.371470 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f\" (UniqueName: \"kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f\") pod \"147865af-a377-4379-8f42-c4d1d622ce91\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " Nov 03 19:42:45.371665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.371650 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca\") pod \"147865af-a377-4379-8f42-c4d1d622ce91\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " Nov 03 19:42:45.371839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.371828 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca\") pod \"147865af-a377-4379-8f42-c4d1d622ce91\" (UID: \"147865af-a377-4379-8f42-c4d1d622ce91\") " Nov 03 19:42:45.375276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.375243 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca" pod "147865af-a377-4379-8f42-c4d1d622ce91" (UID: "147865af-a377-4379-8f42-c4d1d622ce91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:45.375870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.375848 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca" pod "147865af-a377-4379-8f42-c4d1d622ce91" (UID: "147865af-a377-4379-8f42-c4d1d622ce91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:45.378003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.377964 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f" pod "147865af-a377-4379-8f42-c4d1d622ce91" (UID: "147865af-a377-4379-8f42-c4d1d622ce91"). InnerVolumeSpecName "kube-api-access-z9r6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:45.433680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.433650 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:45.474270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.474237 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9r6f\" (UniqueName: \"kubernetes.io/projected/147865af-a377-4379-8f42-c4d1d622ce91-kube-api-access-z9r6f\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:45.474270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.474273 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:45.474449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.474289 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147865af-a377-4379-8f42-c4d1d622ce91-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:45.575334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.575256 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca\") pod \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " Nov 03 19:42:45.575476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.575348 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca\") pod \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " Nov 03 19:42:45.575476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.575473 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88\" (UniqueName: \"kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88\") pod \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\" (UID: \"eb30f743-6e82-4fa6-be71-7c79ee23aa1e\") " Nov 03 19:42:45.575677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.575654 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca" pod "eb30f743-6e82-4fa6-be71-7c79ee23aa1e" (UID: "eb30f743-6e82-4fa6-be71-7c79ee23aa1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:45.575962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.575945 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:45.579018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.578045 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca" pod "eb30f743-6e82-4fa6-be71-7c79ee23aa1e" (UID: "eb30f743-6e82-4fa6-be71-7c79ee23aa1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:45.579018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.578104 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88" pod "eb30f743-6e82-4fa6-be71-7c79ee23aa1e" (UID: "eb30f743-6e82-4fa6-be71-7c79ee23aa1e"). InnerVolumeSpecName "kube-api-access-pxs88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:45.677420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.677388 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:45.677563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:45.677435 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxs88\" (UniqueName: \"kubernetes.io/projected/eb30f743-6e82-4fa6-be71-7c79ee23aa1e-kube-api-access-pxs88\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:46.051369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.051334 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" event={"ID":"eb30f743-6e82-4fa6-be71-7c79ee23aa1e","Type":"ContainerDied","Data":"c1ea57cae3f19fbc7734e2f8d20c593c05be839190a29de51334f0cf072236fd"} Nov 03 19:42:46.051618 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.051602 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ea57cae3f19fbc7734e2f8d20c593c05be839190a29de51334f0cf072236fd" Nov 03 19:42:46.051770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.051759 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8" Nov 03 19:42:46.054071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.054042 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" event={"ID":"147865af-a377-4379-8f42-c4d1d622ce91","Type":"ContainerDied","Data":"aaa00a27c6a001230cc6ddc45893480066123f1f4c7ccad3c0196d6fc20867ea"} Nov 03 19:42:46.054261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.054245 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaa00a27c6a001230cc6ddc45893480066123f1f4c7ccad3c0196d6fc20867ea" Nov 03 19:42:46.054407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.054396 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2" Nov 03 19:42:46.243801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.243768 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:46.244009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:46.243822 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:47.244616 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:47.244574 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:42:47.245144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:47.244644 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:42:48.263239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:48.261642 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:48.263770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:48.263464 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:48.267607 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:48.267574 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:42:49.551891 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.551855 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4"] Nov 03 19:42:49.552540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.552525 5233 cpu_manager.go:414] "containerMap: removing container" podUID="147865af-a377-4379-8f42-c4d1d622ce91" containerName="revoke-keys" Nov 03 19:42:49.552587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.552541 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147865af-a377-4379-8f42-c4d1d622ce91" containerName="revoke-keys" Nov 03 19:42:49.552587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.552570 5233 cpu_manager.go:414] "containerMap: removing container" podUID="eb30f743-6e82-4fa6-be71-7c79ee23aa1e" containerName="revoke-keys" Nov 03 19:42:49.552587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.552575 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb30f743-6e82-4fa6-be71-7c79ee23aa1e" containerName="revoke-keys" Nov 03 19:42:49.552738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.552727 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb30f743-6e82-4fa6-be71-7c79ee23aa1e" containerName="revoke-keys" Nov 03 19:42:49.552773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.552747 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="147865af-a377-4379-8f42-c4d1d622ce91" containerName="revoke-keys" Nov 03 19:42:49.557021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.556990 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.564180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.564158 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:42:49.564441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.564425 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:42:49.567816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.567787 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4"] Nov 03 19:42:49.668863 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.668824 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.669185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.669158 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.669298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.669275 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vhm\" (UniqueName: \"kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.771346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.771312 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.771495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.771383 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vhm\" (UniqueName: \"kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.771495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.771475 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.771959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.771938 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.773779 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.773762 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.784120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.784097 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vhm\" (UniqueName: \"kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm\") pod \"maas-api-revoke-sub-59c22b6e-fkzz4\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:49.885506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:49.885418 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:50.265662 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:50.265631 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627ceecc_efa8_4556_a6bf_c332db936f47.slice/crio-0ad04d556893bd1272790d7f8a7c5d8bf054c0a17187d7c5ef2375b62364f56a WatchSource:0}: Error finding container 0ad04d556893bd1272790d7f8a7c5d8bf054c0a17187d7c5ef2375b62364f56a: Status 404 returned error can't find the container with id 0ad04d556893bd1272790d7f8a7c5d8bf054c0a17187d7c5ef2375b62364f56a Nov 03 19:42:50.269856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.269826 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4"] Nov 03 19:42:50.422609 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.422576 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght"] Nov 03 19:42:50.428848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.428822 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.441023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.437458 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght"] Nov 03 19:42:50.590565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.590494 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.590908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.590665 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.590908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.590693 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fknjf\" (UniqueName: \"kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.692272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.692243 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.692451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.692410 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.692451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.692437 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fknjf\" (UniqueName: \"kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.693192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.693120 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.694784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.694760 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.726897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.726865 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fknjf\" (UniqueName: \"kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf\") pod \"maas-api-revoke-sub-5c2b3c69-glght\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:50.752372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:50.752347 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:51.059390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:51.059358 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght"] Nov 03 19:42:51.065570 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:51.065543 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd32916_c92f_4ca6_aac9_69763db76076.slice/crio-9635722fd268977af2fd008d312ca5d3b59e3856f675cdf1c5439f66e96f2485 WatchSource:0}: Error finding container 9635722fd268977af2fd008d312ca5d3b59e3856f675cdf1c5439f66e96f2485: Status 404 returned error can't find the container with id 9635722fd268977af2fd008d312ca5d3b59e3856f675cdf1c5439f66e96f2485 Nov 03 19:42:51.118062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:51.118027 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" event={"ID":"627ceecc-efa8-4556-a6bf-c332db936f47","Type":"ContainerStarted","Data":"92dd494d499c386d09e80667b959573b27fbd9d2909708a44ba791d6d57dba9a"} Nov 03 19:42:51.118191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:51.118085 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" event={"ID":"627ceecc-efa8-4556-a6bf-c332db936f47","Type":"ContainerStarted","Data":"0ad04d556893bd1272790d7f8a7c5d8bf054c0a17187d7c5ef2375b62364f56a"} Nov 03 19:42:51.119556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:51.119536 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" event={"ID":"4cd32916-c92f-4ca6-aac9-69763db76076","Type":"ContainerStarted","Data":"9635722fd268977af2fd008d312ca5d3b59e3856f675cdf1c5439f66e96f2485"} Nov 03 19:42:51.145755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:51.145705 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" podStartSLOduration=2.145688604 podStartE2EDuration="2.145688604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:51.139336107 +0000 UTC m=+937.480449591" watchObservedRunningTime="2026-11-03 19:42:51.145688604 +0000 UTC m=+937.486802089" Nov 03 19:42:51.313470 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:42:51.313441 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:42:52.130900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:52.130862 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" event={"ID":"4cd32916-c92f-4ca6-aac9-69763db76076","Type":"ContainerStarted","Data":"1d13ba10e99558714ac6fae98f0bd3c79ec90875a0d654dc70a5309c4947876b"} Nov 03 19:42:52.132903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:52.132871 5233 generic.go:356] "Generic (PLEG): container finished" podID="627ceecc-efa8-4556-a6bf-c332db936f47" containerID="92dd494d499c386d09e80667b959573b27fbd9d2909708a44ba791d6d57dba9a" exitCode=0 Nov 03 19:42:52.133075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:52.132913 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" event={"ID":"627ceecc-efa8-4556-a6bf-c332db936f47","Type":"ContainerDied","Data":"92dd494d499c386d09e80667b959573b27fbd9d2909708a44ba791d6d57dba9a"} Nov 03 19:42:52.156681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:52.156634 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" podStartSLOduration=2.156620292 podStartE2EDuration="2.156620292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:52.155006184 +0000 UTC m=+938.496119670" watchObservedRunningTime="2026-11-03 19:42:52.156620292 +0000 UTC m=+938.497733776" Nov 03 19:42:53.143051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.143019 5233 generic.go:356] "Generic (PLEG): container finished" podID="4cd32916-c92f-4ca6-aac9-69763db76076" containerID="1d13ba10e99558714ac6fae98f0bd3c79ec90875a0d654dc70a5309c4947876b" exitCode=0 Nov 03 19:42:53.143486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.143108 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" event={"ID":"4cd32916-c92f-4ca6-aac9-69763db76076","Type":"ContainerDied","Data":"1d13ba10e99558714ac6fae98f0bd3c79ec90875a0d654dc70a5309c4947876b"} Nov 03 19:42:53.331411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.331387 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:53.451153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.451046 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca\") pod \"627ceecc-efa8-4556-a6bf-c332db936f47\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " Nov 03 19:42:53.451153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.451148 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca\") pod \"627ceecc-efa8-4556-a6bf-c332db936f47\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " Nov 03 19:42:53.451327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.451274 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm\" (UniqueName: \"kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm\") pod \"627ceecc-efa8-4556-a6bf-c332db936f47\" (UID: \"627ceecc-efa8-4556-a6bf-c332db936f47\") " Nov 03 19:42:53.451488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.451457 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca" pod "627ceecc-efa8-4556-a6bf-c332db936f47" (UID: "627ceecc-efa8-4556-a6bf-c332db936f47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:53.451942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.451924 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:53.453256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.453213 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca" pod "627ceecc-efa8-4556-a6bf-c332db936f47" (UID: "627ceecc-efa8-4556-a6bf-c332db936f47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:53.453421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.453396 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm" pod "627ceecc-efa8-4556-a6bf-c332db936f47" (UID: "627ceecc-efa8-4556-a6bf-c332db936f47"). InnerVolumeSpecName "kube-api-access-k6vhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:53.553597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.553568 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627ceecc-efa8-4556-a6bf-c332db936f47-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:53.553597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:53.553595 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vhm\" (UniqueName: \"kubernetes.io/projected/627ceecc-efa8-4556-a6bf-c332db936f47-kube-api-access-k6vhm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:54.112181 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.112091 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm"] Nov 03 19:42:54.112823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.112805 5233 cpu_manager.go:414] "containerMap: removing container" podUID="627ceecc-efa8-4556-a6bf-c332db936f47" containerName="revoke-keys" Nov 03 19:42:54.112823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.112823 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627ceecc-efa8-4556-a6bf-c332db936f47" containerName="revoke-keys" Nov 03 19:42:54.112989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.112960 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="627ceecc-efa8-4556-a6bf-c332db936f47" containerName="revoke-keys" Nov 03 19:42:54.117034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.117008 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.130209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.130184 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm"] Nov 03 19:42:54.154318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.154283 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" Nov 03 19:42:54.154318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.154310 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4" event={"ID":"627ceecc-efa8-4556-a6bf-c332db936f47","Type":"ContainerDied","Data":"0ad04d556893bd1272790d7f8a7c5d8bf054c0a17187d7c5ef2375b62364f56a"} Nov 03 19:42:54.154803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.154362 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad04d556893bd1272790d7f8a7c5d8bf054c0a17187d7c5ef2375b62364f56a" Nov 03 19:42:54.165835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.165808 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.165952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.165838 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz44\" (UniqueName: \"kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.165952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.165872 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.267679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.267640 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.268079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.268057 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.268160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.268093 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz44\" (UniqueName: \"kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.268930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.268906 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.270996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.270904 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.281288 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.281262 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz44\" (UniqueName: \"kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44\") pod \"maas-api-revoke-sub-39e28e5c-q6rqm\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.347416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.347386 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:54.369123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.369039 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf\" (UniqueName: \"kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf\") pod \"4cd32916-c92f-4ca6-aac9-69763db76076\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " Nov 03 19:42:54.369324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.369205 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca\") pod \"4cd32916-c92f-4ca6-aac9-69763db76076\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " Nov 03 19:42:54.369324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.369264 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca\") pod \"4cd32916-c92f-4ca6-aac9-69763db76076\" (UID: \"4cd32916-c92f-4ca6-aac9-69763db76076\") " Nov 03 19:42:54.369782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.369753 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca" pod "4cd32916-c92f-4ca6-aac9-69763db76076" (UID: "4cd32916-c92f-4ca6-aac9-69763db76076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:54.369935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.369916 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:54.371346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.371324 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf" pod "4cd32916-c92f-4ca6-aac9-69763db76076" (UID: "4cd32916-c92f-4ca6-aac9-69763db76076"). InnerVolumeSpecName "kube-api-access-fknjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:54.372174 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.372068 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca" pod "4cd32916-c92f-4ca6-aac9-69763db76076" (UID: "4cd32916-c92f-4ca6-aac9-69763db76076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:54.442550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.442523 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:54.471806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.471782 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd32916-c92f-4ca6-aac9-69763db76076-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:54.471916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.471809 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fknjf\" (UniqueName: \"kubernetes.io/projected/4cd32916-c92f-4ca6-aac9-69763db76076-kube-api-access-fknjf\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:54.486228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.486197 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7"] Nov 03 19:42:54.487257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.487237 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4cd32916-c92f-4ca6-aac9-69763db76076" containerName="revoke-keys" Nov 03 19:42:54.487354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.487258 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd32916-c92f-4ca6-aac9-69763db76076" containerName="revoke-keys" Nov 03 19:42:54.487448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.487436 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd32916-c92f-4ca6-aac9-69763db76076" containerName="revoke-keys" Nov 03 19:42:54.512625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.512598 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7"] Nov 03 19:42:54.515138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.512910 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.573108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.573070 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.573276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.573255 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrrg\" (UniqueName: \"kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.573444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.573412 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.674560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.674470 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.674735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.674579 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrrg\" (UniqueName: \"kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.674735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.674638 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.675307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.675285 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.676807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.676781 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.688588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.688560 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrrg\" (UniqueName: \"kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg\") pod \"maas-api-revoke-sub-f1594089-44gs7\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:54.750093 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:54.750061 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea7d8dd_cb5b_44ca_8128_2820db2e5453.slice/crio-469e4cf6add06b5db3e23abf1dc2b1a4b71e504fe9db6f8af16a4bd70f3a4a6b WatchSource:0}: Error finding container 469e4cf6add06b5db3e23abf1dc2b1a4b71e504fe9db6f8af16a4bd70f3a4a6b: Status 404 returned error can't find the container with id 469e4cf6add06b5db3e23abf1dc2b1a4b71e504fe9db6f8af16a4bd70f3a4a6b Nov 03 19:42:54.752114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.752090 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm"] Nov 03 19:42:54.850882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:54.850835 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:55.134113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.134081 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n"] Nov 03 19:42:55.140328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.140298 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.146585 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.146558 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n"] Nov 03 19:42:55.167421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.167338 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" Nov 03 19:42:55.167421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.167355 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght" event={"ID":"4cd32916-c92f-4ca6-aac9-69763db76076","Type":"ContainerDied","Data":"9635722fd268977af2fd008d312ca5d3b59e3856f675cdf1c5439f66e96f2485"} Nov 03 19:42:55.167421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.167396 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9635722fd268977af2fd008d312ca5d3b59e3856f675cdf1c5439f66e96f2485" Nov 03 19:42:55.169319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.169293 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" event={"ID":"5ea7d8dd-cb5b-44ca-8128-2820db2e5453","Type":"ContainerStarted","Data":"469e4cf6add06b5db3e23abf1dc2b1a4b71e504fe9db6f8af16a4bd70f3a4a6b"} Nov 03 19:42:55.182461 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:55.182436 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5262db5c_5e88_47a6_8d97_5dff20a54984.slice/crio-b165083e54909d65764b048fbb5cc496d6e4cdc0e2b2916b5360621be62ca499 WatchSource:0}: Error finding container b165083e54909d65764b048fbb5cc496d6e4cdc0e2b2916b5360621be62ca499: Status 404 returned error can't find the container with id b165083e54909d65764b048fbb5cc496d6e4cdc0e2b2916b5360621be62ca499 Nov 03 19:42:55.182564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.182509 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmxl\" (UniqueName: \"kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.182628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.182604 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.182675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.182666 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.184216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.184196 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7"] Nov 03 19:42:55.283706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.283679 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh"] Nov 03 19:42:55.289370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.287447 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.289370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.287567 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.289370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.287779 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmxl\" (UniqueName: \"kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.289370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.288919 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.323168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.315225 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.323168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.323054 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.336767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.334639 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh"] Nov 03 19:42:55.386760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.386729 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmxl\" (UniqueName: \"kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl\") pod \"maas-api-revoke-sub-a3d1ab60-bxq2n\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.389114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389083 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389244 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389220 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389274 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389446 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389426 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389498 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tvs\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389601 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389586 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389632 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389697 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.389771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.389722 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.458981 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.458953 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:55.491556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491519 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.491709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491581 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.491709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491647 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.491709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491671 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.491860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491766 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.491860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491795 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.491860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.491844 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.492064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.492045 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.492132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.492117 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tvs\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.493095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.493006 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.493732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.493703 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.494019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.493957 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.494269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.494248 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.494896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.494870 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.496124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.496099 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.499795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.499771 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.507995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.504544 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.507995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.505802 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tvs\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs\") pod \"e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.741895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.741866 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:55.862423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:55.861946 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n"] Nov 03 19:42:56.063196 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:42:56.063165 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3133c63_b11f_4d87_88ce_c94c5f1efd13.slice/crio-c15b5cf08c1b32ab4295c7ca740768f321fa69bdcb36545b3dc55e644af9dd5d WatchSource:0}: Error finding container c15b5cf08c1b32ab4295c7ca740768f321fa69bdcb36545b3dc55e644af9dd5d: Status 404 returned error can't find the container with id c15b5cf08c1b32ab4295c7ca740768f321fa69bdcb36545b3dc55e644af9dd5d Nov 03 19:42:56.065008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.064953 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh"] Nov 03 19:42:56.066199 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.066170 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:56.066250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.066233 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:56.066280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.066262 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:42:56.183355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.183318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" event={"ID":"5262db5c-5e88-47a6-8d97-5dff20a54984","Type":"ContainerStarted","Data":"ad701871e46c8c5b53ba521760cc978faef67e8b2504fce6576072a1acb9aed9"} Nov 03 19:42:56.183783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.183448 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" event={"ID":"5262db5c-5e88-47a6-8d97-5dff20a54984","Type":"ContainerStarted","Data":"b165083e54909d65764b048fbb5cc496d6e4cdc0e2b2916b5360621be62ca499"} Nov 03 19:42:56.186724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.186506 5233 generic.go:356] "Generic (PLEG): container finished" podID="5ea7d8dd-cb5b-44ca-8128-2820db2e5453" containerID="4e4332303c3c8ff642083e3f28b73a31a9dc77819f38bfb7c2161bd31df84940" exitCode=0 Nov 03 19:42:56.186724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.186584 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" event={"ID":"5ea7d8dd-cb5b-44ca-8128-2820db2e5453","Type":"ContainerDied","Data":"4e4332303c3c8ff642083e3f28b73a31a9dc77819f38bfb7c2161bd31df84940"} Nov 03 19:42:56.188952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.188928 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" event={"ID":"625e5f15-958c-4d23-aae7-43b58adbddfb","Type":"ContainerStarted","Data":"67500c0c01b24438921b07374cc616111a1a85ce413caee8e41c17b3901250f0"} Nov 03 19:42:56.198610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.197700 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" event={"ID":"c3133c63-b11f-4d87-88ce-c94c5f1efd13","Type":"ContainerStarted","Data":"2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0"} Nov 03 19:42:56.198610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.197757 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" event={"ID":"c3133c63-b11f-4d87-88ce-c94c5f1efd13","Type":"ContainerStarted","Data":"c15b5cf08c1b32ab4295c7ca740768f321fa69bdcb36545b3dc55e644af9dd5d"} Nov 03 19:42:56.237685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.234123 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" podStartSLOduration=2.234103047 podStartE2EDuration="2.234103047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:56.206568723 +0000 UTC m=+942.547682210" watchObservedRunningTime="2026-11-03 19:42:56.234103047 +0000 UTC m=+942.575216534" Nov 03 19:42:56.271123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.271072 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podStartSLOduration=1.27105618 podStartE2EDuration="1.27105618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:56.243531218 +0000 UTC m=+942.584644704" watchObservedRunningTime="2026-11-03 19:42:56.27105618 +0000 UTC m=+942.612169667" Nov 03 19:42:56.758052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:56.744378 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:42:57.231359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.231321 5233 generic.go:356] "Generic (PLEG): container finished" podID="5262db5c-5e88-47a6-8d97-5dff20a54984" containerID="ad701871e46c8c5b53ba521760cc978faef67e8b2504fce6576072a1acb9aed9" exitCode=0 Nov 03 19:42:57.231765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.231397 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" event={"ID":"5262db5c-5e88-47a6-8d97-5dff20a54984","Type":"ContainerDied","Data":"ad701871e46c8c5b53ba521760cc978faef67e8b2504fce6576072a1acb9aed9"} Nov 03 19:42:57.233416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.233382 5233 generic.go:356] "Generic (PLEG): container finished" podID="625e5f15-958c-4d23-aae7-43b58adbddfb" containerID="52bbe6b56d296dedbb60232f18c226f4f17e1c09ccd3fa073f942571a7076aeb" exitCode=0 Nov 03 19:42:57.233558 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.233531 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" event={"ID":"625e5f15-958c-4d23-aae7-43b58adbddfb","Type":"ContainerDied","Data":"52bbe6b56d296dedbb60232f18c226f4f17e1c09ccd3fa073f942571a7076aeb"} Nov 03 19:42:57.593834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.593712 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:57.657996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.655303 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44\" (UniqueName: \"kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44\") pod \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " Nov 03 19:42:57.657996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.655428 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca\") pod \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " Nov 03 19:42:57.657996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.655718 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca\") pod \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\" (UID: \"5ea7d8dd-cb5b-44ca-8128-2820db2e5453\") " Nov 03 19:42:57.658447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.658292 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca" pod "5ea7d8dd-cb5b-44ca-8128-2820db2e5453" (UID: "5ea7d8dd-cb5b-44ca-8128-2820db2e5453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:57.665994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.661117 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44" pod "5ea7d8dd-cb5b-44ca-8128-2820db2e5453" (UID: "5ea7d8dd-cb5b-44ca-8128-2820db2e5453"). InnerVolumeSpecName "kube-api-access-hcz44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:57.668909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.666803 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca" pod "5ea7d8dd-cb5b-44ca-8128-2820db2e5453" (UID: "5ea7d8dd-cb5b-44ca-8128-2820db2e5453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:57.743676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.743635 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:42:57.743836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.743697 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:42:57.757421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.757390 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:57.757421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.757417 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:57.757596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:57.757426 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcz44\" (UniqueName: \"kubernetes.io/projected/5ea7d8dd-cb5b-44ca-8128-2820db2e5453-kube-api-access-hcz44\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:58.246215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.246180 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" Nov 03 19:42:58.246215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.246197 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm" event={"ID":"5ea7d8dd-cb5b-44ca-8128-2820db2e5453","Type":"ContainerDied","Data":"469e4cf6add06b5db3e23abf1dc2b1a4b71e504fe9db6f8af16a4bd70f3a4a6b"} Nov 03 19:42:58.246795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.246246 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469e4cf6add06b5db3e23abf1dc2b1a4b71e504fe9db6f8af16a4bd70f3a4a6b" Nov 03 19:42:58.588079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.572371 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:58.671862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.671827 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:58.674892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.674860 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca\") pod \"625e5f15-958c-4d23-aae7-43b58adbddfb\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " Nov 03 19:42:58.675494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.675465 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl\" (UniqueName: \"kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl\") pod \"625e5f15-958c-4d23-aae7-43b58adbddfb\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " Nov 03 19:42:58.675657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.675637 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca\") pod \"625e5f15-958c-4d23-aae7-43b58adbddfb\" (UID: \"625e5f15-958c-4d23-aae7-43b58adbddfb\") " Nov 03 19:42:58.677174 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.677144 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca" pod "625e5f15-958c-4d23-aae7-43b58adbddfb" (UID: "625e5f15-958c-4d23-aae7-43b58adbddfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:58.678845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.678818 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca" pod "625e5f15-958c-4d23-aae7-43b58adbddfb" (UID: "625e5f15-958c-4d23-aae7-43b58adbddfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:58.679161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.679133 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl" pod "625e5f15-958c-4d23-aae7-43b58adbddfb" (UID: "625e5f15-958c-4d23-aae7-43b58adbddfb"). InnerVolumeSpecName "kube-api-access-xkmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:58.742954 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.742913 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:42:58.743163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.742998 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:42:58.778150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.778116 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca\") pod \"5262db5c-5e88-47a6-8d97-5dff20a54984\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " Nov 03 19:42:58.778393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.778164 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg\" (UniqueName: \"kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg\") pod \"5262db5c-5e88-47a6-8d97-5dff20a54984\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " Nov 03 19:42:58.778393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.778259 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca\") pod \"5262db5c-5e88-47a6-8d97-5dff20a54984\" (UID: \"5262db5c-5e88-47a6-8d97-5dff20a54984\") " Nov 03 19:42:58.778688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.778671 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmxl\" (UniqueName: \"kubernetes.io/projected/625e5f15-958c-4d23-aae7-43b58adbddfb-kube-api-access-xkmxl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:58.778688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.778688 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:58.778843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.778699 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625e5f15-958c-4d23-aae7-43b58adbddfb-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:58.779227 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.779203 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca" pod "5262db5c-5e88-47a6-8d97-5dff20a54984" (UID: "5262db5c-5e88-47a6-8d97-5dff20a54984"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:58.780347 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.780320 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca" pod "5262db5c-5e88-47a6-8d97-5dff20a54984" (UID: "5262db5c-5e88-47a6-8d97-5dff20a54984"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:42:58.782800 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.782767 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg" pod "5262db5c-5e88-47a6-8d97-5dff20a54984" (UID: "5262db5c-5e88-47a6-8d97-5dff20a54984"). InnerVolumeSpecName "kube-api-access-mgrrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:58.880703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.880618 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:58.880703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.880688 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgrrg\" (UniqueName: \"kubernetes.io/projected/5262db5c-5e88-47a6-8d97-5dff20a54984-kube-api-access-mgrrg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:58.880882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:58.880707 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5262db5c-5e88-47a6-8d97-5dff20a54984-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:59.258546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.258516 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" Nov 03 19:42:59.259058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.258549 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n" event={"ID":"625e5f15-958c-4d23-aae7-43b58adbddfb","Type":"ContainerDied","Data":"67500c0c01b24438921b07374cc616111a1a85ce413caee8e41c17b3901250f0"} Nov 03 19:42:59.259058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.258601 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67500c0c01b24438921b07374cc616111a1a85ce413caee8e41c17b3901250f0" Nov 03 19:42:59.261173 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.261129 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" Nov 03 19:42:59.261432 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.261130 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7" event={"ID":"5262db5c-5e88-47a6-8d97-5dff20a54984","Type":"ContainerDied","Data":"b165083e54909d65764b048fbb5cc496d6e4cdc0e2b2916b5360621be62ca499"} Nov 03 19:42:59.261589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.261571 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b165083e54909d65764b048fbb5cc496d6e4cdc0e2b2916b5360621be62ca499" Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.682329 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx"] Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.684669 5233 cpu_manager.go:414] "containerMap: removing container" podUID="625e5f15-958c-4d23-aae7-43b58adbddfb" containerName="revoke-keys" Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.684713 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625e5f15-958c-4d23-aae7-43b58adbddfb" containerName="revoke-keys" Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.684766 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5262db5c-5e88-47a6-8d97-5dff20a54984" containerName="revoke-keys" Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.684776 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5262db5c-5e88-47a6-8d97-5dff20a54984" containerName="revoke-keys" Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.684837 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5ea7d8dd-cb5b-44ca-8128-2820db2e5453" containerName="revoke-keys" Nov 03 19:42:59.684840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.684846 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea7d8dd-cb5b-44ca-8128-2820db2e5453" containerName="revoke-keys" Nov 03 19:42:59.685418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.685390 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="625e5f15-958c-4d23-aae7-43b58adbddfb" containerName="revoke-keys" Nov 03 19:42:59.685476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.685429 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5262db5c-5e88-47a6-8d97-5dff20a54984" containerName="revoke-keys" Nov 03 19:42:59.685476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.685442 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea7d8dd-cb5b-44ca-8128-2820db2e5453" containerName="revoke-keys" Nov 03 19:42:59.739262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.739216 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.742455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.742423 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:42:59.743116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.743044 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:42:59.743778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.743750 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx"] Nov 03 19:42:59.743894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.743833 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:42:59.744052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.743899 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:42:59.840366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.840333 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.840579 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.840516 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.840794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.840776 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnscd\" (UniqueName: \"kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.942629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.942544 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.942788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.942653 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.942870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.942852 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnscd\" (UniqueName: \"kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.943903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.943874 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.945648 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.945622 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:42:59.955670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:42:59.955647 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnscd\" (UniqueName: \"kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd\") pod \"maas-api-revoke-sub-e8621181-qwhlx\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:43:00.061745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:00.061712 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:43:00.671100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:00.671048 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx"] Nov 03 19:43:00.744474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:00.743959 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:00.744474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:00.744056 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:00.815154 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:00.815117 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdad68d5_09d7_43ee_9120_cf5d27dbb80e.slice/crio-68dbb31379f5eeac302e5edcb6093cf6cdbc2d04baecff5a076d359ca63c0f92 WatchSource:0}: Error finding container 68dbb31379f5eeac302e5edcb6093cf6cdbc2d04baecff5a076d359ca63c0f92: Status 404 returned error can't find the container with id 68dbb31379f5eeac302e5edcb6093cf6cdbc2d04baecff5a076d359ca63c0f92 Nov 03 19:43:01.211101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.209049 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d"] Nov 03 19:43:01.459399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.451358 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d"] Nov 03 19:43:01.459399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.451488 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.534435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.533943 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.534435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.534075 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.534435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.534187 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47gb\" (UniqueName: \"kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.621078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.613681 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" event={"ID":"cdad68d5-09d7-43ee-9120-cf5d27dbb80e","Type":"ContainerStarted","Data":"68dbb31379f5eeac302e5edcb6093cf6cdbc2d04baecff5a076d359ca63c0f92"} Nov 03 19:43:01.642097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.638665 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.642097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.638796 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47gb\" (UniqueName: \"kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.642097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.639049 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.642097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.640104 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.691924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.691888 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.743998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.740003 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47gb\" (UniqueName: \"kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb\") pod \"maas-api-revoke-sub-d84c6e31-5xr4d\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:01.753994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.750382 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:01.753994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.750440 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:01.811284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.810076 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:01.967771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:01.967730 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:02.059010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.051481 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:02.059010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.052273 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:02.059010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.052925 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:02.524998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.493123 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:02.549998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.540069 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:02.575013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.566987 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:02.844996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.794136 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:02.847115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.847075 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:02.847689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.811613 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:02.848273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.848252 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:43:02.889992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.889188 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 19:43:02.983132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:02.983080 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:43:03.056133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.056068 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:43:03.064652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.064621 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:43:03.144053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.064884 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:03.144053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.064918 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:03.144053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.120146 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:43:03.144053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.120211 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:43:03.511538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.511495 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:03.568814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.530468 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:43:03.573650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.573608 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" event={"ID":"cdad68d5-09d7-43ee-9120-cf5d27dbb80e","Type":"ContainerStarted","Data":"767ac36a518179340b8f6c603a746c4298f7e6f496775a405396a20a1b912a6f"} Nov 03 19:43:03.574450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.546113 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:03.590138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.579894 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:43:03.626025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.622147 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:03.626025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.622305 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:43:03.974009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.955879 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:03.976276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.975110 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:03.976276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:03.975205 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:04.141984 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.121151 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:04.199516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.199453 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:43:04.274113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.255069 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:04.381547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.378408 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:43:04.517234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.517188 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qvdpc" podUID="92edadb3-9fae-48a3-aed1-54b25cf7c0ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 19:43:04.601054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.538953 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:04.705922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.705878 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:04.709993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.668331 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:04.787675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.787620 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:43:05.018064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.998999 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:05.018064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:04.999084 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.042144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.042105 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:05.104794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.104751 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.140922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.140881 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:05.164314 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.145413 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:43:05.190002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.160207 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.190467 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.173012 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:43:05.202148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.202096 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.237997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.235107 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:05.237997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.235236 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.311218 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.311105 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.439106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.423081 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.439106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.423187 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded" Nov 03 19:43:05.439106 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.423245 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 19:43:05.545995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.542409 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:05.783998 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:43:05.755370 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.673s" Nov 03 19:43:05.783998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.755543 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:43:05.783998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.757451 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:05.783998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.778363 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:05.783998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.778418 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:05.946145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.946099 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:05.983998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.959803 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:05.983998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:05.959867 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.178994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.154407 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:06.178994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.154474 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.180176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.180142 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:06.180380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.180355 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.191574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.191528 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.191794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.191769 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:43:06.215421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.215370 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.578143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.578099 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.681995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.666794 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:06.746748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.746706 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.775997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.762103 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:06.809997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.798348 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:06.809997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.798406 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:06.811315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.811274 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" event={"ID":"4cb34026-7268-4264-8e3c-420b52c65890","Type":"ContainerStarted","Data":"4bdc0a5fe6508d0581d2fc23167bf3e6db67f4917d3160989ebff709b044fc56"} Nov 03 19:43:06.899239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.899145 5233 generic.go:356] "Generic (PLEG): container finished" podID="cdad68d5-09d7-43ee-9120-cf5d27dbb80e" containerID="767ac36a518179340b8f6c603a746c4298f7e6f496775a405396a20a1b912a6f" exitCode=0 Nov 03 19:43:06.903235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.903193 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" event={"ID":"cdad68d5-09d7-43ee-9120-cf5d27dbb80e","Type":"ContainerDied","Data":"767ac36a518179340b8f6c603a746c4298f7e6f496775a405396a20a1b912a6f"} Nov 03 19:43:06.975016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:06.972383 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.206764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.206658 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.208754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.208706 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:43:07.219994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.215788 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.307121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.307061 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d"] Nov 03 19:43:07.613074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.609554 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:43:07.692735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.692692 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:07.770028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.758099 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:07.770028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.758161 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:07.770028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.760059 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.770028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.763490 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.813997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.811037 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52"] Nov 03 19:43:07.914077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.909467 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:07.914077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.909528 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.940476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.940115 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:07.940476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.940179 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:07.990679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:07.980424 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:08.118487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.089089 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:08.221113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.221054 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:43:08.227499 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.227452 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:43:08.274239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.234418 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:08.275227 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.270254 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" event={"ID":"4cb34026-7268-4264-8e3c-420b52c65890","Type":"ContainerStarted","Data":"43eb5e0f4574ada090d449eb5fc299849456818862b73da4bf917199bc412a6f"} Nov 03 19:43:08.278736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.270367 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.357943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.355847 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" podStartSLOduration=7.355825721 podStartE2EDuration="7.355825721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:08.269864798 +0000 UTC m=+954.610978283" watchObservedRunningTime="2026-11-03 19:43:08.355825721 +0000 UTC m=+954.696939207" Nov 03 19:43:08.498044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.497936 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.498331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.498291 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsxl\" (UniqueName: \"kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.506103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.506058 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.628404 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.628459 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsxl\" (UniqueName: \"kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.628657 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.629837 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.630316 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.668420 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.713634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.687712 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsxl\" (UniqueName: \"kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl\") pod \"maas-api-revoke-sub-29a3d46c-mcf52\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:08.816057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:08.738472 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:09.097393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.096148 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:43:09.169998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.103842 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:09.169998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.103909 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:09.169998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.147372 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:09.227997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.223403 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:09.227997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.223483 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.243903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.239464 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 19:43:09.267995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.245087 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.292998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.262800 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.330996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.321078 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.330996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.321988 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.330996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.322313 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.649055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.640676 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:09.775275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.775242 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 19:43:09.775551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.775520 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 19:43:09.786209 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:43:09.786177 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.511s" Nov 03 19:43:09.786522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.786505 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52"] Nov 03 19:43:09.786764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.786749 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:43:09.843118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:09.843092 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:43:10.107115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.105694 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:10.120741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.112916 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:43:10.120741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.115302 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:43:10.120741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.118792 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" Nov 03 19:43:10.129754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.126471 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx" event={"ID":"cdad68d5-09d7-43ee-9120-cf5d27dbb80e","Type":"ContainerDied","Data":"68dbb31379f5eeac302e5edcb6093cf6cdbc2d04baecff5a076d359ca63c0f92"} Nov 03 19:43:10.129754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.126521 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68dbb31379f5eeac302e5edcb6093cf6cdbc2d04baecff5a076d359ca63c0f92" Nov 03 19:43:10.129754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.126563 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:43:10.129754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.128554 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:43:10.144581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.144557 5233 generic.go:356] "Generic (PLEG): container finished" podID="4cb34026-7268-4264-8e3c-420b52c65890" containerID="43eb5e0f4574ada090d449eb5fc299849456818862b73da4bf917199bc412a6f" exitCode=0 Nov 03 19:43:10.145024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.144936 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" event={"ID":"4cb34026-7268-4264-8e3c-420b52c65890","Type":"ContainerDied","Data":"43eb5e0f4574ada090d449eb5fc299849456818862b73da4bf917199bc412a6f"} Nov 03 19:43:10.145200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.145187 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:43:10.176052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.176021 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd\" (UniqueName: \"kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd\") pod \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " Nov 03 19:43:10.176359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.176344 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca\") pod \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " Nov 03 19:43:10.176497 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.176483 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca\") pod \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\" (UID: \"cdad68d5-09d7-43ee-9120-cf5d27dbb80e\") " Nov 03 19:43:10.180447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.180423 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca" pod "cdad68d5-09d7-43ee-9120-cf5d27dbb80e" (UID: "cdad68d5-09d7-43ee-9120-cf5d27dbb80e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:10.182003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.181663 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:43:10.195047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.193455 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd" pod "cdad68d5-09d7-43ee-9120-cf5d27dbb80e" (UID: "cdad68d5-09d7-43ee-9120-cf5d27dbb80e"). InnerVolumeSpecName "kube-api-access-mnscd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:10.195047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.193876 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:43:10.236752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.236675 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca" pod "cdad68d5-09d7-43ee-9120-cf5d27dbb80e" (UID: "cdad68d5-09d7-43ee-9120-cf5d27dbb80e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:10.280067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.280038 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnscd\" (UniqueName: \"kubernetes.io/projected/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-kube-api-access-mnscd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:10.280067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.280067 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:10.280067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.280083 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdad68d5-09d7-43ee-9120-cf5d27dbb80e-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:10.408207 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:10.408101 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218c7627_7a47_4f87_b520_a04f66eaa43a.slice/crio-6beb68cf649dbb2e9c92fcf16b6d3345ccf636337518012e992bcc69addf74be WatchSource:0}: Error finding container 6beb68cf649dbb2e9c92fcf16b6d3345ccf636337518012e992bcc69addf74be: Status 404 returned error can't find the container with id 6beb68cf649dbb2e9c92fcf16b6d3345ccf636337518012e992bcc69addf74be Nov 03 19:43:10.409040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.408763 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52"] Nov 03 19:43:10.743997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:10.743612 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:43:11.096351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.096130 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:43:11.162477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.162439 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" event={"ID":"218c7627-7a47-4f87-b520-a04f66eaa43a","Type":"ContainerStarted","Data":"5d7eddb4f012b6664c2e1a9ae4cf8432a968a0c9820c9bf1a1c477768f44cb79"} Nov 03 19:43:11.162666 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.162507 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" event={"ID":"218c7627-7a47-4f87-b520-a04f66eaa43a","Type":"ContainerStarted","Data":"6beb68cf649dbb2e9c92fcf16b6d3345ccf636337518012e992bcc69addf74be"} Nov 03 19:43:11.164168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.164148 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:43:11.195993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.192662 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" podStartSLOduration=5.192643335 podStartE2EDuration="5.192643335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:11.181492422 +0000 UTC m=+957.522605907" watchObservedRunningTime="2026-11-03 19:43:11.192643335 +0000 UTC m=+957.533756821" Nov 03 19:43:11.442262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.442067 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:11.544373 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.544179 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb\" (UniqueName: \"kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb\") pod \"4cb34026-7268-4264-8e3c-420b52c65890\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " Nov 03 19:43:11.550303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.546639 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca\") pod \"4cb34026-7268-4264-8e3c-420b52c65890\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " Nov 03 19:43:11.550303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.546806 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca\") pod \"4cb34026-7268-4264-8e3c-420b52c65890\" (UID: \"4cb34026-7268-4264-8e3c-420b52c65890\") " Nov 03 19:43:11.550303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.547492 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca" pod "4cb34026-7268-4264-8e3c-420b52c65890" (UID: "4cb34026-7268-4264-8e3c-420b52c65890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:11.550303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.548025 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:11.550646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.550623 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca" pod "4cb34026-7268-4264-8e3c-420b52c65890" (UID: "4cb34026-7268-4264-8e3c-420b52c65890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:11.550691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.550641 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb" pod "4cb34026-7268-4264-8e3c-420b52c65890" (UID: "4cb34026-7268-4264-8e3c-420b52c65890"). InnerVolumeSpecName "kube-api-access-k47gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:11.650019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.649911 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47gb\" (UniqueName: \"kubernetes.io/projected/4cb34026-7268-4264-8e3c-420b52c65890-kube-api-access-k47gb\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:11.650247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:11.650228 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb34026-7268-4264-8e3c-420b52c65890-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:12.175080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.175049 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" Nov 03 19:43:12.175294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.175214 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d" event={"ID":"4cb34026-7268-4264-8e3c-420b52c65890","Type":"ContainerDied","Data":"4bdc0a5fe6508d0581d2fc23167bf3e6db67f4917d3160989ebff709b044fc56"} Nov 03 19:43:12.175294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.175267 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bdc0a5fe6508d0581d2fc23167bf3e6db67f4917d3160989ebff709b044fc56" Nov 03 19:43:12.183011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.180852 5233 generic.go:356] "Generic (PLEG): container finished" podID="218c7627-7a47-4f87-b520-a04f66eaa43a" containerID="5d7eddb4f012b6664c2e1a9ae4cf8432a968a0c9820c9bf1a1c477768f44cb79" exitCode=0 Nov 03 19:43:12.183011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.180933 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" event={"ID":"218c7627-7a47-4f87-b520-a04f66eaa43a","Type":"ContainerDied","Data":"5d7eddb4f012b6664c2e1a9ae4cf8432a968a0c9820c9bf1a1c477768f44cb79"} Nov 03 19:43:12.318088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.318053 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7"] Nov 03 19:43:12.319252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.319224 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cdad68d5-09d7-43ee-9120-cf5d27dbb80e" containerName="revoke-keys" Nov 03 19:43:12.319252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.319250 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdad68d5-09d7-43ee-9120-cf5d27dbb80e" containerName="revoke-keys" Nov 03 19:43:12.319458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.319322 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4cb34026-7268-4264-8e3c-420b52c65890" containerName="revoke-keys" Nov 03 19:43:12.319458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.319330 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb34026-7268-4264-8e3c-420b52c65890" containerName="revoke-keys" Nov 03 19:43:12.319559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.319516 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb34026-7268-4264-8e3c-420b52c65890" containerName="revoke-keys" Nov 03 19:43:12.319559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.319539 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdad68d5-09d7-43ee-9120-cf5d27dbb80e" containerName="revoke-keys" Nov 03 19:43:12.324352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.324330 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.351743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.351714 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7"] Nov 03 19:43:12.361462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.361437 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbgwf\" (UniqueName: \"kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.361572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.361549 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.361661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.361647 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.463481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.463449 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.463934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.463704 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbgwf\" (UniqueName: \"kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.463934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.463802 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.464544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.464524 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.465836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.465813 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.480678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.480652 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbgwf\" (UniqueName: \"kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf\") pod \"maas-api-revoke-sub-1150360e-5tdw7\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.650047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.650014 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:12.978692 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:12.978665 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b842fb1_0d91_4240_b380_f0e085faf9e5.slice/crio-942a894609e8e5870da6cd5d2e177214a06e7d237c0c7411e7a211de2bc0a490 WatchSource:0}: Error finding container 942a894609e8e5870da6cd5d2e177214a06e7d237c0c7411e7a211de2bc0a490: Status 404 returned error can't find the container with id 942a894609e8e5870da6cd5d2e177214a06e7d237c0c7411e7a211de2bc0a490 Nov 03 19:43:12.980214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:12.980185 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7"] Nov 03 19:43:13.203938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.203888 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" event={"ID":"8b842fb1-0d91-4240-b380-f0e085faf9e5","Type":"ContainerStarted","Data":"942a894609e8e5870da6cd5d2e177214a06e7d237c0c7411e7a211de2bc0a490"} Nov 03 19:43:13.435422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.435396 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:13.492059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.492031 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca\") pod \"218c7627-7a47-4f87-b520-a04f66eaa43a\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " Nov 03 19:43:13.492500 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.492168 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca\") pod \"218c7627-7a47-4f87-b520-a04f66eaa43a\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " Nov 03 19:43:13.492500 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.492240 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl\" (UniqueName: \"kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl\") pod \"218c7627-7a47-4f87-b520-a04f66eaa43a\" (UID: \"218c7627-7a47-4f87-b520-a04f66eaa43a\") " Nov 03 19:43:13.493222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.493192 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca" pod "218c7627-7a47-4f87-b520-a04f66eaa43a" (UID: "218c7627-7a47-4f87-b520-a04f66eaa43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:13.494685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.494656 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl" pod "218c7627-7a47-4f87-b520-a04f66eaa43a" (UID: "218c7627-7a47-4f87-b520-a04f66eaa43a"). InnerVolumeSpecName "kube-api-access-9nsxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:13.496085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.496051 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca" pod "218c7627-7a47-4f87-b520-a04f66eaa43a" (UID: "218c7627-7a47-4f87-b520-a04f66eaa43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:13.594598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.594514 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:13.594598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.594544 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nsxl\" (UniqueName: \"kubernetes.io/projected/218c7627-7a47-4f87-b520-a04f66eaa43a-kube-api-access-9nsxl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:13.594598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:13.594554 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218c7627-7a47-4f87-b520-a04f66eaa43a-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:14.231001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:14.230610 5233 generic.go:356] "Generic (PLEG): container finished" podID="8b842fb1-0d91-4240-b380-f0e085faf9e5" containerID="6db9159dbc26e161e494c7cfd3456092e09fb48484c279179ea123fe54a8d9c4" exitCode=0 Nov 03 19:43:14.231001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:14.230948 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" event={"ID":"8b842fb1-0d91-4240-b380-f0e085faf9e5","Type":"ContainerDied","Data":"6db9159dbc26e161e494c7cfd3456092e09fb48484c279179ea123fe54a8d9c4"} Nov 03 19:43:14.233390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:14.233362 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" event={"ID":"218c7627-7a47-4f87-b520-a04f66eaa43a","Type":"ContainerDied","Data":"6beb68cf649dbb2e9c92fcf16b6d3345ccf636337518012e992bcc69addf74be"} Nov 03 19:43:14.233512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:14.233410 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6beb68cf649dbb2e9c92fcf16b6d3345ccf636337518012e992bcc69addf74be" Nov 03 19:43:14.233512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:14.233469 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52" Nov 03 19:43:15.422144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.422111 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:15.526314 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.526228 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca\") pod \"8b842fb1-0d91-4240-b380-f0e085faf9e5\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " Nov 03 19:43:15.526469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.526431 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf\" (UniqueName: \"kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf\") pod \"8b842fb1-0d91-4240-b380-f0e085faf9e5\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " Nov 03 19:43:15.526544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.526503 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca\") pod \"8b842fb1-0d91-4240-b380-f0e085faf9e5\" (UID: \"8b842fb1-0d91-4240-b380-f0e085faf9e5\") " Nov 03 19:43:15.527590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.527534 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca" pod "8b842fb1-0d91-4240-b380-f0e085faf9e5" (UID: "8b842fb1-0d91-4240-b380-f0e085faf9e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:15.543802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.532872 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:15.543802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.541356 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca" pod "8b842fb1-0d91-4240-b380-f0e085faf9e5" (UID: "8b842fb1-0d91-4240-b380-f0e085faf9e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:15.545177 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.545147 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf" pod "8b842fb1-0d91-4240-b380-f0e085faf9e5" (UID: "8b842fb1-0d91-4240-b380-f0e085faf9e5"). InnerVolumeSpecName "kube-api-access-pbgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:15.634667 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.634636 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbgwf\" (UniqueName: \"kubernetes.io/projected/8b842fb1-0d91-4240-b380-f0e085faf9e5-kube-api-access-pbgwf\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:15.634667 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:15.634665 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b842fb1-0d91-4240-b380-f0e085faf9e5-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:16.253302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:16.253272 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" Nov 03 19:43:16.253503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:16.253311 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7" event={"ID":"8b842fb1-0d91-4240-b380-f0e085faf9e5","Type":"ContainerDied","Data":"942a894609e8e5870da6cd5d2e177214a06e7d237c0c7411e7a211de2bc0a490"} Nov 03 19:43:16.253503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:16.253358 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="942a894609e8e5870da6cd5d2e177214a06e7d237c0c7411e7a211de2bc0a490" Nov 03 19:43:18.805573 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.805540 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst"] Nov 03 19:43:18.806357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.806340 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8b842fb1-0d91-4240-b380-f0e085faf9e5" containerName="revoke-keys" Nov 03 19:43:18.806430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.806358 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b842fb1-0d91-4240-b380-f0e085faf9e5" containerName="revoke-keys" Nov 03 19:43:18.806430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.806395 5233 cpu_manager.go:414] "containerMap: removing container" podUID="218c7627-7a47-4f87-b520-a04f66eaa43a" containerName="revoke-keys" Nov 03 19:43:18.806430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.806402 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218c7627-7a47-4f87-b520-a04f66eaa43a" containerName="revoke-keys" Nov 03 19:43:18.806529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.806513 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="218c7627-7a47-4f87-b520-a04f66eaa43a" containerName="revoke-keys" Nov 03 19:43:18.806529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.806528 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b842fb1-0d91-4240-b380-f0e085faf9e5" containerName="revoke-keys" Nov 03 19:43:18.813749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.813712 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.831942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.831913 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ca12e9fb-gw-openshift-default-dockercfg-28v8m\"" Nov 03 19:43:18.838957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.838932 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst"] Nov 03 19:43:18.920760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.920724 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.920961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.920806 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.920961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.920924 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.921096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.920985 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.921136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.921117 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvpr\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.921179 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.921153 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.921231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.921189 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.921351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.921331 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:18.921420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:18.921386 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.038992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.038461 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.043996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.043478 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.055428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.055040 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.055428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.055171 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.055827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.055768 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.055915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.055845 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.055915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.055879 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.056043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.056026 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvpr\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.056093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.056071 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.056156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.056137 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.061987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.056663 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.061987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.057041 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.061987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.058944 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.061987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.059234 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.072860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.072834 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.075961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.075899 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.079080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.079016 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvpr\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.084243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.083754 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token\") pod \"e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.140001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.138852 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:19.503678 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:19.503647 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc61436c_04d0_4dbf_aa81_4ea479a5160e.slice/crio-fc3f18bc12ebd00cdb7dc730bddb9f0028e289290ec5c9805d055c887a080043 WatchSource:0}: Error finding container fc3f18bc12ebd00cdb7dc730bddb9f0028e289290ec5c9805d055c887a080043: Status 404 returned error can't find the container with id fc3f18bc12ebd00cdb7dc730bddb9f0028e289290ec5c9805d055c887a080043 Nov 03 19:43:19.506589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.506554 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:19.506709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.506637 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:19.506709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.506680 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:19.506709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.506683 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst"] Nov 03 19:43:19.636991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.630712 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz"] Nov 03 19:43:19.636991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.636718 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.654636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.654592 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz"] Nov 03 19:43:19.668012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.657377 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:43:19.668012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.662604 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:43:19.668310 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.668285 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7np7q\" (UniqueName: \"kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.668379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.668363 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.668908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.668799 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.770265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.770176 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.770457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.770439 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.770529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.770511 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7np7q\" (UniqueName: \"kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.771410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.771354 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.773853 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.773771 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.783308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.783278 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7np7q\" (UniqueName: \"kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q\") pod \"maas-api-revoke-sub-1afcc4c7-2nrhz\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:19.967308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:19.967172 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:20.046644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.045175 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb"] Nov 03 19:43:20.054234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.054204 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.060768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.059441 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb"] Nov 03 19:43:20.178332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.178304 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.178471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.178459 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvq2\" (UniqueName: \"kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.178562 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.178550 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.279487 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:20.279453 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386e90a1_4f59_4eaf_9913_035edce32dad.slice/crio-ee9c3899171e3e7bc3fe8c3a73220062f7dd682d375ee8d8e5208acaccfaec4e WatchSource:0}: Error finding container ee9c3899171e3e7bc3fe8c3a73220062f7dd682d375ee8d8e5208acaccfaec4e: Status 404 returned error can't find the container with id ee9c3899171e3e7bc3fe8c3a73220062f7dd682d375ee8d8e5208acaccfaec4e Nov 03 19:43:20.279789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.279765 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvq2\" (UniqueName: \"kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.279897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.279882 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.280571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.280069 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.280961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.280936 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.281466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.281388 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz"] Nov 03 19:43:20.283216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.283197 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.303756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.303737 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvq2\" (UniqueName: \"kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2\") pod \"maas-api-revoke-sub-eb4a982f-rrfsb\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.351286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.351251 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" event={"ID":"fc61436c-04d0-4dbf-aa81-4ea479a5160e","Type":"ContainerStarted","Data":"fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a"} Nov 03 19:43:20.351429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.351343 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" event={"ID":"fc61436c-04d0-4dbf-aa81-4ea479a5160e","Type":"ContainerStarted","Data":"fc3f18bc12ebd00cdb7dc730bddb9f0028e289290ec5c9805d055c887a080043"} Nov 03 19:43:20.353332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.353308 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" event={"ID":"386e90a1-4f59-4eaf-9913-035edce32dad","Type":"ContainerStarted","Data":"ee9c3899171e3e7bc3fe8c3a73220062f7dd682d375ee8d8e5208acaccfaec4e"} Nov 03 19:43:20.380562 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.380510 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" podStartSLOduration=2.380492001 podStartE2EDuration="2.380492001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:20.377533677 +0000 UTC m=+966.718647163" watchObservedRunningTime="2026-11-03 19:43:20.380492001 +0000 UTC m=+966.721605488" Nov 03 19:43:20.424149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.424121 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:20.777841 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:20.777801 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b2eda3_01bb_4529_a93c_11475c5a11af.slice/crio-67142455788a1ff9471375481296c6dd534b9f34c1eb6f2d86f1257af14bcc5a WatchSource:0}: Error finding container 67142455788a1ff9471375481296c6dd534b9f34c1eb6f2d86f1257af14bcc5a: Status 404 returned error can't find the container with id 67142455788a1ff9471375481296c6dd534b9f34c1eb6f2d86f1257af14bcc5a Nov 03 19:43:20.779709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:20.779680 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb"] Nov 03 19:43:21.140926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.140843 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:21.146935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.146910 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:21.280014 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:43:21.279962 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:43:21.364611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.364575 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" event={"ID":"70b2eda3-01bb-4529-a93c-11475c5a11af","Type":"ContainerStarted","Data":"5ba7d7574c558e0ff8a7f6c26a4be0933262fd714a4e4126296309cf6244254f"} Nov 03 19:43:21.364794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.364633 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" event={"ID":"70b2eda3-01bb-4529-a93c-11475c5a11af","Type":"ContainerStarted","Data":"67142455788a1ff9471375481296c6dd534b9f34c1eb6f2d86f1257af14bcc5a"} Nov 03 19:43:21.366438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.366410 5233 generic.go:356] "Generic (PLEG): container finished" podID="386e90a1-4f59-4eaf-9913-035edce32dad" containerID="c63cb1c75fe575bae745df8dcd2b4fdd6bbc9edcb84d410d0da5008db19fe30a" exitCode=0 Nov 03 19:43:21.366554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.366486 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" event={"ID":"386e90a1-4f59-4eaf-9913-035edce32dad","Type":"ContainerDied","Data":"c63cb1c75fe575bae745df8dcd2b4fdd6bbc9edcb84d410d0da5008db19fe30a"} Nov 03 19:43:21.366846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.366830 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:21.367882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.367865 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:43:21.389595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:21.389554 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" podStartSLOduration=1.389541062 podStartE2EDuration="1.389541062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:21.383465876 +0000 UTC m=+967.724579363" watchObservedRunningTime="2026-11-03 19:43:21.389541062 +0000 UTC m=+967.730654546" Nov 03 19:43:22.378445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.378413 5233 generic.go:356] "Generic (PLEG): container finished" podID="70b2eda3-01bb-4529-a93c-11475c5a11af" containerID="5ba7d7574c558e0ff8a7f6c26a4be0933262fd714a4e4126296309cf6244254f" exitCode=0 Nov 03 19:43:22.378959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.378512 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" event={"ID":"70b2eda3-01bb-4529-a93c-11475c5a11af","Type":"ContainerDied","Data":"5ba7d7574c558e0ff8a7f6c26a4be0933262fd714a4e4126296309cf6244254f"} Nov 03 19:43:22.558785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.558758 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:22.621835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.621800 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q\" (UniqueName: \"kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q\") pod \"386e90a1-4f59-4eaf-9913-035edce32dad\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " Nov 03 19:43:22.622082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.622060 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca\") pod \"386e90a1-4f59-4eaf-9913-035edce32dad\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " Nov 03 19:43:22.622170 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.622109 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca\") pod \"386e90a1-4f59-4eaf-9913-035edce32dad\" (UID: \"386e90a1-4f59-4eaf-9913-035edce32dad\") " Nov 03 19:43:22.622539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.622511 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca" pod "386e90a1-4f59-4eaf-9913-035edce32dad" (UID: "386e90a1-4f59-4eaf-9913-035edce32dad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:22.622747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.622732 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:22.624072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.624044 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q" pod "386e90a1-4f59-4eaf-9913-035edce32dad" (UID: "386e90a1-4f59-4eaf-9913-035edce32dad"). InnerVolumeSpecName "kube-api-access-7np7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:22.624155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.624114 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca" pod "386e90a1-4f59-4eaf-9913-035edce32dad" (UID: "386e90a1-4f59-4eaf-9913-035edce32dad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:22.725842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.724193 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386e90a1-4f59-4eaf-9913-035edce32dad-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:22.725842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.724227 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7np7q\" (UniqueName: \"kubernetes.io/projected/386e90a1-4f59-4eaf-9913-035edce32dad-kube-api-access-7np7q\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:22.893744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.893657 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47"] Nov 03 19:43:22.894467 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.894446 5233 cpu_manager.go:414] "containerMap: removing container" podUID="386e90a1-4f59-4eaf-9913-035edce32dad" containerName="revoke-keys" Nov 03 19:43:22.894467 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.894466 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386e90a1-4f59-4eaf-9913-035edce32dad" containerName="revoke-keys" Nov 03 19:43:22.894738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.894724 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="386e90a1-4f59-4eaf-9913-035edce32dad" containerName="revoke-keys" Nov 03 19:43:22.898961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.898940 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:22.904256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.904230 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ca12e9fb\"" Nov 03 19:43:22.904852 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.904825 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ca12e9fb-dockercfg-nf4q8\"" Nov 03 19:43:22.913516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.913488 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47"] Nov 03 19:43:22.929739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.928041 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw"] Nov 03 19:43:22.937822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:22.937791 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.015644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.015609 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw"] Nov 03 19:43:23.161049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.055071 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.161049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.080910 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d728n\" (UniqueName: \"kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n\") pod \"payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.161049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.081038 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6k7\" (UniqueName: \"kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7\") pod \"payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.161049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.081193 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.193994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.188918 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d728n\" (UniqueName: \"kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n\") pod \"payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.193994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.189082 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6k7\" (UniqueName: \"kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7\") pod \"payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.193994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.189521 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.193994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.189622 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.193994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.190767 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.253236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.251806 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d728n\" (UniqueName: \"kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n\") pod \"payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.253236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.252603 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.272010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.271904 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6k7\" (UniqueName: \"kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7\") pod \"payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.365376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.365342 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:23.391053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.391027 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" Nov 03 19:43:23.391411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.391045 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz" event={"ID":"386e90a1-4f59-4eaf-9913-035edce32dad","Type":"ContainerDied","Data":"ee9c3899171e3e7bc3fe8c3a73220062f7dd682d375ee8d8e5208acaccfaec4e"} Nov 03 19:43:23.391411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.391106 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9c3899171e3e7bc3fe8c3a73220062f7dd682d375ee8d8e5208acaccfaec4e" Nov 03 19:43:23.522186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.521666 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:23.732372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.728886 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw"] Nov 03 19:43:23.742591 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:23.742542 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8236266_058e_429d_a8b4_89e376265d5a.slice/crio-2c2e6a5c2da0990152f713deec47a4a08a87378f40c85708a80c0e1653a80be1 WatchSource:0}: Error finding container 2c2e6a5c2da0990152f713deec47a4a08a87378f40c85708a80c0e1653a80be1: Status 404 returned error can't find the container with id 2c2e6a5c2da0990152f713deec47a4a08a87378f40c85708a80c0e1653a80be1 Nov 03 19:43:23.761313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.761291 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:23.935423 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:23.935382 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9c03c1_f5c6_4b0c_b59b_96927570e2a3.slice/crio-adbddbec0f1c105bca956b08ff99b4238f79aa7a6cf1e2a14718477e79dcaa37 WatchSource:0}: Error finding container adbddbec0f1c105bca956b08ff99b4238f79aa7a6cf1e2a14718477e79dcaa37: Status 404 returned error can't find the container with id adbddbec0f1c105bca956b08ff99b4238f79aa7a6cf1e2a14718477e79dcaa37 Nov 03 19:43:23.937789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.937768 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47"] Nov 03 19:43:23.944041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.944018 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca\") pod \"70b2eda3-01bb-4529-a93c-11475c5a11af\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " Nov 03 19:43:23.944145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.944125 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca\") pod \"70b2eda3-01bb-4529-a93c-11475c5a11af\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " Nov 03 19:43:23.944274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.944262 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2\" (UniqueName: \"kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2\") pod \"70b2eda3-01bb-4529-a93c-11475c5a11af\" (UID: \"70b2eda3-01bb-4529-a93c-11475c5a11af\") " Nov 03 19:43:23.944351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.944331 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca" pod "70b2eda3-01bb-4529-a93c-11475c5a11af" (UID: "70b2eda3-01bb-4529-a93c-11475c5a11af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:23.944862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.944841 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:23.946459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.946435 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca" pod "70b2eda3-01bb-4529-a93c-11475c5a11af" (UID: "70b2eda3-01bb-4529-a93c-11475c5a11af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:23.946795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:23.946773 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2" pod "70b2eda3-01bb-4529-a93c-11475c5a11af" (UID: "70b2eda3-01bb-4529-a93c-11475c5a11af"). InnerVolumeSpecName "kube-api-access-jcvq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:24.046949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.046917 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2eda3-01bb-4529-a93c-11475c5a11af-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:24.046949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.046944 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvq2\" (UniqueName: \"kubernetes.io/projected/70b2eda3-01bb-4529-a93c-11475c5a11af-kube-api-access-jcvq2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:24.148491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.148459 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr"] Nov 03 19:43:24.149588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.149457 5233 cpu_manager.go:414] "containerMap: removing container" podUID="70b2eda3-01bb-4529-a93c-11475c5a11af" containerName="revoke-keys" Nov 03 19:43:24.149588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.149492 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b2eda3-01bb-4529-a93c-11475c5a11af" containerName="revoke-keys" Nov 03 19:43:24.149832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.149728 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b2eda3-01bb-4529-a93c-11475c5a11af" containerName="revoke-keys" Nov 03 19:43:24.154387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.154354 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.163418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.163389 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr"] Nov 03 19:43:24.250819 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.250725 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.250984 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.250942 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.251078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.251059 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglq4\" (UniqueName: \"kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.352755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.352715 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.352941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.352862 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.352941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.352919 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglq4\" (UniqueName: \"kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.353420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.353400 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.355191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.355169 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.368444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.368424 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglq4\" (UniqueName: \"kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4\") pod \"maas-api-revoke-sub-7290e24c-9jrbr\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.403051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.403022 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" event={"ID":"70b2eda3-01bb-4529-a93c-11475c5a11af","Type":"ContainerDied","Data":"67142455788a1ff9471375481296c6dd534b9f34c1eb6f2d86f1257af14bcc5a"} Nov 03 19:43:24.403415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.403079 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67142455788a1ff9471375481296c6dd534b9f34c1eb6f2d86f1257af14bcc5a" Nov 03 19:43:24.403415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.403090 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb" Nov 03 19:43:24.405054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.405027 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" event={"ID":"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3","Type":"ContainerStarted","Data":"ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d"} Nov 03 19:43:24.405175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.405082 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" event={"ID":"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3","Type":"ContainerStarted","Data":"adbddbec0f1c105bca956b08ff99b4238f79aa7a6cf1e2a14718477e79dcaa37"} Nov 03 19:43:24.405175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.405133 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:24.407340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.407318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" event={"ID":"a8236266-058e-429d-a8b4-89e376265d5a","Type":"ContainerStarted","Data":"0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e"} Nov 03 19:43:24.407441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.407361 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" event={"ID":"a8236266-058e-429d-a8b4-89e376265d5a","Type":"ContainerStarted","Data":"2c2e6a5c2da0990152f713deec47a4a08a87378f40c85708a80c0e1653a80be1"} Nov 03 19:43:24.407441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.407389 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:24.438208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.438159 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" podStartSLOduration=2.43814498 podStartE2EDuration="2.43814498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:24.431269522 +0000 UTC m=+970.772383008" watchObservedRunningTime="2026-11-03 19:43:24.43814498 +0000 UTC m=+970.779258464" Nov 03 19:43:24.474211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.474178 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:24.475560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.475520 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" podStartSLOduration=2.475507432 podStartE2EDuration="2.475507432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:24.468955189 +0000 UTC m=+970.810068677" watchObservedRunningTime="2026-11-03 19:43:24.475507432 +0000 UTC m=+970.816620917" Nov 03 19:43:24.771186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:24.771156 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr"] Nov 03 19:43:24.775591 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:24.775566 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2f44f8_7025_407e_a9c3_a83d7f2af1c2.slice/crio-dee8b74d7615791198fab6176e0f1196554538180c56016076800890fa0d59fb WatchSource:0}: Error finding container dee8b74d7615791198fab6176e0f1196554538180c56016076800890fa0d59fb: Status 404 returned error can't find the container with id dee8b74d7615791198fab6176e0f1196554538180c56016076800890fa0d59fb Nov 03 19:43:25.418792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:25.418696 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" event={"ID":"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2","Type":"ContainerStarted","Data":"982c120d31c4f2bc4ab54feb5a0499c9aa3cded01298eaa711982588b843df99"} Nov 03 19:43:25.419303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:25.418803 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" event={"ID":"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2","Type":"ContainerStarted","Data":"dee8b74d7615791198fab6176e0f1196554538180c56016076800890fa0d59fb"} Nov 03 19:43:25.470827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:25.470767 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" podStartSLOduration=1.47075486 podStartE2EDuration="1.47075486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:25.46060843 +0000 UTC m=+971.801721914" watchObservedRunningTime="2026-11-03 19:43:25.47075486 +0000 UTC m=+971.811868344" Nov 03 19:43:26.451999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.442985 5233 generic.go:356] "Generic (PLEG): container finished" podID="aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" containerID="982c120d31c4f2bc4ab54feb5a0499c9aa3cded01298eaa711982588b843df99" exitCode=0 Nov 03 19:43:26.451999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.443287 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" event={"ID":"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2","Type":"ContainerDied","Data":"982c120d31c4f2bc4ab54feb5a0499c9aa3cded01298eaa711982588b843df99"} Nov 03 19:43:26.465797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.465760 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw"] Nov 03 19:43:26.492995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.484842 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.523906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.523872 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.524184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.524166 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.524549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.524535 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.551476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.548719 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw"] Nov 03 19:43:26.626270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.626238 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.626431 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.626387 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.626515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.626440 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.627147 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.627117 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.629087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.629067 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.651626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.651605 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f\") pod \"maas-api-revoke-sub-6744125b-f9wcw\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:26.846299 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:26.846266 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:27.144453 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:27.144421 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfd816b_4816_40bc_955f_bdb17922f3fa.slice/crio-4aeeabf36afbf8635782022c4a06621452e7e2f1bac8820ee75a569d634684d4 WatchSource:0}: Error finding container 4aeeabf36afbf8635782022c4a06621452e7e2f1bac8820ee75a569d634684d4: Status 404 returned error can't find the container with id 4aeeabf36afbf8635782022c4a06621452e7e2f1bac8820ee75a569d634684d4 Nov 03 19:43:27.146295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:27.146249 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw"] Nov 03 19:43:27.467002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:27.466921 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" event={"ID":"5dfd816b-4816-40bc-955f-bdb17922f3fa","Type":"ContainerStarted","Data":"4aeeabf36afbf8635782022c4a06621452e7e2f1bac8820ee75a569d634684d4"} Nov 03 19:43:27.950691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:27.950653 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:28.085812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.085771 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca\") pod \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " Nov 03 19:43:28.086058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.085840 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca\") pod \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " Nov 03 19:43:28.086058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.085923 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4\" (UniqueName: \"kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4\") pod \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\" (UID: \"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2\") " Nov 03 19:43:28.086571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.086395 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca" pod "aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" (UID: "aa2f44f8-7025-407e-a9c3-a83d7f2af1c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:28.086741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.086680 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:28.088293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.088237 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca" pod "aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" (UID: "aa2f44f8-7025-407e-a9c3-a83d7f2af1c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:28.088928 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.088899 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4" pod "aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" (UID: "aa2f44f8-7025-407e-a9c3-a83d7f2af1c2"). InnerVolumeSpecName "kube-api-access-bglq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:28.188217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.188180 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:28.188217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.188219 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bglq4\" (UniqueName: \"kubernetes.io/projected/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2-kube-api-access-bglq4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:28.481772 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.481735 5233 generic.go:356] "Generic (PLEG): container finished" podID="5dfd816b-4816-40bc-955f-bdb17922f3fa" containerID="12276ab1a6c7eda660e67a42dcf95b1522cbc4a9975d7c1e17dee17a30ffac5f" exitCode=0 Nov 03 19:43:28.482456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.481817 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" event={"ID":"5dfd816b-4816-40bc-955f-bdb17922f3fa","Type":"ContainerDied","Data":"12276ab1a6c7eda660e67a42dcf95b1522cbc4a9975d7c1e17dee17a30ffac5f"} Nov 03 19:43:28.484498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.484473 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" Nov 03 19:43:28.484620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.484527 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr" event={"ID":"aa2f44f8-7025-407e-a9c3-a83d7f2af1c2","Type":"ContainerDied","Data":"dee8b74d7615791198fab6176e0f1196554538180c56016076800890fa0d59fb"} Nov 03 19:43:28.484620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:28.484578 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee8b74d7615791198fab6176e0f1196554538180c56016076800890fa0d59fb" Nov 03 19:43:29.674269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.674244 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:29.715256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.715224 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca\") pod \"5dfd816b-4816-40bc-955f-bdb17922f3fa\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " Nov 03 19:43:29.715428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.715270 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f\") pod \"5dfd816b-4816-40bc-955f-bdb17922f3fa\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " Nov 03 19:43:29.715609 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.715579 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca\") pod \"5dfd816b-4816-40bc-955f-bdb17922f3fa\" (UID: \"5dfd816b-4816-40bc-955f-bdb17922f3fa\") " Nov 03 19:43:29.716179 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.716144 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca" pod "5dfd816b-4816-40bc-955f-bdb17922f3fa" (UID: "5dfd816b-4816-40bc-955f-bdb17922f3fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:29.716454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.716431 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:29.719590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.718255 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca" pod "5dfd816b-4816-40bc-955f-bdb17922f3fa" (UID: "5dfd816b-4816-40bc-955f-bdb17922f3fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:29.719590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.718960 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f" pod "5dfd816b-4816-40bc-955f-bdb17922f3fa" (UID: "5dfd816b-4816-40bc-955f-bdb17922f3fa"). InnerVolumeSpecName "kube-api-access-nfl2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:29.818719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.818618 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfd816b-4816-40bc-955f-bdb17922f3fa-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:29.818719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:29.818660 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/5dfd816b-4816-40bc-955f-bdb17922f3fa-kube-api-access-nfl2f\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:30.514238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:30.514206 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" Nov 03 19:43:30.514238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:30.514222 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw" event={"ID":"5dfd816b-4816-40bc-955f-bdb17922f3fa","Type":"ContainerDied","Data":"4aeeabf36afbf8635782022c4a06621452e7e2f1bac8820ee75a569d634684d4"} Nov 03 19:43:30.514465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:30.514280 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aeeabf36afbf8635782022c4a06621452e7e2f1bac8820ee75a569d634684d4" Nov 03 19:43:32.599704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.599669 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5"] Nov 03 19:43:32.600407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.600393 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5dfd816b-4816-40bc-955f-bdb17922f3fa" containerName="revoke-keys" Nov 03 19:43:32.600456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.600409 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfd816b-4816-40bc-955f-bdb17922f3fa" containerName="revoke-keys" Nov 03 19:43:32.600456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.600447 5233 cpu_manager.go:414] "containerMap: removing container" podUID="aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" containerName="revoke-keys" Nov 03 19:43:32.600456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.600453 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" containerName="revoke-keys" Nov 03 19:43:32.600596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.600585 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfd816b-4816-40bc-955f-bdb17922f3fa" containerName="revoke-keys" Nov 03 19:43:32.600631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.600605 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" containerName="revoke-keys" Nov 03 19:43:32.605242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.605222 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:32.616657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.616634 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-edc935cf\"" Nov 03 19:43:32.619596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.619569 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5"] Nov 03 19:43:32.661261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.661228 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj"] Nov 03 19:43:32.666214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.666195 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:32.681076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.681054 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-edc935cf-dockercfg-j668f\"" Nov 03 19:43:32.692638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.692606 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj"] Nov 03 19:43:32.736231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.735989 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:32.736231 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.736171 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxdl\" (UniqueName: \"kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl\") pod \"payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:32.846997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.840181 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxdl\" (UniqueName: \"kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl\") pod \"payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:32.846997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.840476 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwm9\" (UniqueName: \"kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9\") pod \"payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:32.846997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.840609 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:32.846997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.840768 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:32.905040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.852495 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:32.905860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.905828 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxdl\" (UniqueName: \"kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl\") pod \"payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:33.002991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.946038 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwm9\" (UniqueName: \"kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9\") pod \"payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:33.002991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.946203 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:33.002991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:32.948200 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:33.047635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.047606 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:33.049715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.049654 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwm9\" (UniqueName: \"kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9\") pod \"payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:33.293997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.286680 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:33.376038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.374471 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n"] Nov 03 19:43:33.382176 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:33.382126 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8b0cec_42ec_4bb2_83b2_66092f821b2a.slice/crio-e89053f4e50921ac8aa52e09522b459a03373c36a041766ead415865fb6efd0b WatchSource:0}: Error finding container e89053f4e50921ac8aa52e09522b459a03373c36a041766ead415865fb6efd0b: Status 404 returned error can't find the container with id e89053f4e50921ac8aa52e09522b459a03373c36a041766ead415865fb6efd0b Nov 03 19:43:33.383151 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.382999 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.384215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.383936 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5"] Nov 03 19:43:33.399608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.399446 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:43:33.399608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.399533 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:43:33.400431 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.400399 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n"] Nov 03 19:43:33.464197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.464167 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.464389 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.464248 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9m7\" (UniqueName: \"kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.464389 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.464308 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.556783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.556698 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" event={"ID":"cf8b0cec-42ec-4bb2-83b2-66092f821b2a","Type":"ContainerStarted","Data":"0bf6bfac97d6a32db0fd54732b1fbf29fe982d35817b8036e3e3e1475b9c4b4e"} Nov 03 19:43:33.557045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.557017 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:33.557163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.557049 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" event={"ID":"cf8b0cec-42ec-4bb2-83b2-66092f821b2a","Type":"ContainerStarted","Data":"e89053f4e50921ac8aa52e09522b459a03373c36a041766ead415865fb6efd0b"} Nov 03 19:43:33.565573 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.565545 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.565757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.565733 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9m7\" (UniqueName: \"kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.565859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.565836 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.566442 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.566420 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.568030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.568010 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.579232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.579210 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9m7\" (UniqueName: \"kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7\") pod \"maas-api-revoke-sub-8df54c40-4sr5n\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:33.583592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.583553 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" podStartSLOduration=1.583542427 podStartE2EDuration="1.583542427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:33.57743996 +0000 UTC m=+979.918553445" watchObservedRunningTime="2026-11-03 19:43:33.583542427 +0000 UTC m=+979.924655942" Nov 03 19:43:33.611642 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:33.611614 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e3fd4e_4626_44ab_9406_c2e7c9c92480.slice/crio-9888acf4a754ef27e838ccb819cbd5de3f910fa89e0625c5796b54da8a9abeaf WatchSource:0}: Error finding container 9888acf4a754ef27e838ccb819cbd5de3f910fa89e0625c5796b54da8a9abeaf: Status 404 returned error can't find the container with id 9888acf4a754ef27e838ccb819cbd5de3f910fa89e0625c5796b54da8a9abeaf Nov 03 19:43:33.614164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.614143 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj"] Nov 03 19:43:33.714908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:33.714883 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:34.573997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.570674 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n"] Nov 03 19:43:34.713990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.713876 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" event={"ID":"e3829410-2108-4749-8b62-d32ee82b7e99","Type":"ContainerStarted","Data":"9d17410a236077fecaebfc2fa896f0df04c5dfc964348f91fe47727e59639658"} Nov 03 19:43:34.733168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.733132 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" event={"ID":"64e3fd4e-4626-44ab-9406-c2e7c9c92480","Type":"ContainerStarted","Data":"6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12"} Nov 03 19:43:34.733438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.733414 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" event={"ID":"64e3fd4e-4626-44ab-9406-c2e7c9c92480","Type":"ContainerStarted","Data":"9888acf4a754ef27e838ccb819cbd5de3f910fa89e0625c5796b54da8a9abeaf"} Nov 03 19:43:34.733604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.733589 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:34.825934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.825861 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" podStartSLOduration=2.825841015 podStartE2EDuration="2.825841015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:34.820252516 +0000 UTC m=+981.161366004" watchObservedRunningTime="2026-11-03 19:43:34.825841015 +0000 UTC m=+981.166954501" Nov 03 19:43:34.968158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:34.968114 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:43:35.255055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.232954 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:35.488014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.484877 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:43:35.499322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.498233 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:43:35.878208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.863291 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:43:35.902994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.889458 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:35.953119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.953055 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:35.971112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:35.971054 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:36.250017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:36.244081 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:43:36.275005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:36.274060 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:36.374177 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:36.285990 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:36.374177 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:36.286646 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:37.198885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.124441 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:43:37.287999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.202523 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:37.496997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.296111 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:37.496997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.296430 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:37.515996 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:43:37.299792 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.405s" Nov 03 19:43:37.515996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.507703 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:43:37.543995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.361047 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:43:37.573430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.465090 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:37.624009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.619578 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:43:37.646055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.624637 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:43:37.677018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.656353 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:43:37.677018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.656465 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:43:37.677018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.656525 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:43:37.677018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:37.656579 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:43:38.204002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.138183 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:38.267083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.226212 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:43:38.267666 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.226289 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:38.306244 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.306197 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 19:43:38.428795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.428756 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:38.429144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.429108 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:38.520016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.489418 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:38.525717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.525673 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:38.528182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.528152 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:38.641499 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.638609 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:38.644482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.638647 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:43:39.077107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.977073 5233 patch_prober.go:29] interesting pod/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:39.077107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:38.977134 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.078220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.078186 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:39.078446 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.078420 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.135712 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:43:39.135674 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.305s" Nov 03 19:43:39.136139 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.136116 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:43:39.159129 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.159096 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:43:39.159393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.159366 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" event={"ID":"e3829410-2108-4749-8b62-d32ee82b7e99","Type":"ContainerStarted","Data":"17401bf2bd8261b4f2ca0db461804675cb02b511e4155c405dfec6625c2d7e36"} Nov 03 19:43:39.283001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.271344 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.318052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.287687 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.318052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.310123 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.503070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.479585 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.507541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.506163 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.535223 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.535181 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:43:39.574604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.574559 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:43:39.574991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.574880 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.700487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.700445 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:39.701161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.701130 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:39.701540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:39.701518 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:43:40.141997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.141846 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts"] Nov 03 19:43:40.169064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.169026 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts"] Nov 03 19:43:40.185998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.172550 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.295010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.265425 5233 generic.go:356] "Generic (PLEG): container finished" podID="e3829410-2108-4749-8b62-d32ee82b7e99" containerID="17401bf2bd8261b4f2ca0db461804675cb02b511e4155c405dfec6625c2d7e36" exitCode=0 Nov 03 19:43:40.295010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.266546 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" event={"ID":"e3829410-2108-4749-8b62-d32ee82b7e99","Type":"ContainerDied","Data":"17401bf2bd8261b4f2ca0db461804675cb02b511e4155c405dfec6625c2d7e36"} Nov 03 19:43:40.295010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.282821 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:43:40.295010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.288059 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:43:40.380570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.380510 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:40.458031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.457916 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Nov 03 19:43:40.555221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.555186 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:40.559300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.555141 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:40.596286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.596117 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:40.686025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.667572 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl4q\" (UniqueName: \"kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.686025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.667734 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.686025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.667889 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.736472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.736429 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:40.737116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.736710 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:43:40.737736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.736738 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:43:40.772323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.772287 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl4q\" (UniqueName: \"kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.790201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.790170 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.790595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.790569 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.802994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.791576 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:40.933926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:40.933870 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl"] Nov 03 19:43:41.081495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.081382 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.264995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.226422 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:41.357045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.353175 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xknwm\" (UniqueName: \"kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.357045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.353313 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.357045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.353428 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.383459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.374056 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:41.444201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.444162 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl4q\" (UniqueName: \"kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q\") pod \"maas-api-revoke-sub-60d261ab-zwmts\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:41.449089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.449050 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:41.582283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.462285 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:43:41.582283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.473554 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknwm\" (UniqueName: \"kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.582283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.473658 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.582283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.473749 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.582283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.479556 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.582283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.481514 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:41.800329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.800278 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:41.813649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.813585 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xknwm\" (UniqueName: \"kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.831018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.823191 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:43:41.831018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.824273 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:43:41.831018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.825001 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:43:41.834845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.834819 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:41.839043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.835652 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:43:41.885931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.879524 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl"] Nov 03 19:43:41.885931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.879664 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:43:41.892290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.892254 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67f295a-hxjkl\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:41.969630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.966105 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:43:41.969630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.968792 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:43:41.971048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.971002 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:43:41.973999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.973505 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:43:41.976026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.975785 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:43:41.979926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:41.979910 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:42.024159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.024129 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2"] Nov 03 19:43:42.025652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.025611 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e3829410-2108-4749-8b62-d32ee82b7e99" containerName="revoke-keys" Nov 03 19:43:42.025904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.025861 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3829410-2108-4749-8b62-d32ee82b7e99" containerName="revoke-keys" Nov 03 19:43:42.026951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.026930 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3829410-2108-4749-8b62-d32ee82b7e99" containerName="revoke-keys" Nov 03 19:43:42.036828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.032429 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.036828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.033743 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2"] Nov 03 19:43:42.039242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.039167 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7\" (UniqueName: \"kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7\") pod \"e3829410-2108-4749-8b62-d32ee82b7e99\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " Nov 03 19:43:42.040343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.040322 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca\") pod \"e3829410-2108-4749-8b62-d32ee82b7e99\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " Nov 03 19:43:42.040752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.040733 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca\") pod \"e3829410-2108-4749-8b62-d32ee82b7e99\" (UID: \"e3829410-2108-4749-8b62-d32ee82b7e99\") " Nov 03 19:43:42.043871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.043812 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7" pod "e3829410-2108-4749-8b62-d32ee82b7e99" (UID: "e3829410-2108-4749-8b62-d32ee82b7e99"). InnerVolumeSpecName "kube-api-access-lk9m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:42.045178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.044872 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca" pod "e3829410-2108-4749-8b62-d32ee82b7e99" (UID: "e3829410-2108-4749-8b62-d32ee82b7e99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:42.048830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.048802 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca" pod "e3829410-2108-4749-8b62-d32ee82b7e99" (UID: "e3829410-2108-4749-8b62-d32ee82b7e99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:42.143586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143466 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.143586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143576 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.143910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143654 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.143910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143833 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.143910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143890 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.143910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143914 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.144189 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.143994 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqndq\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.144189 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.144103 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.144290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.144275 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.144390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.144375 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9m7\" (UniqueName: \"kubernetes.io/projected/e3829410-2108-4749-8b62-d32ee82b7e99-kube-api-access-lk9m7\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:42.144460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.144395 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:42.144460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.144411 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3829410-2108-4749-8b62-d32ee82b7e99-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:42.166890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.166861 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:42.219788 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:42.219742 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98a9b0a_1de9_4a96_95b7_82a72f313b19.slice/crio-c3b4b0922cf30339f6ce5bfe524cf7702d47af06b7608b0df0a26e785bdfcdc4 WatchSource:0}: Error finding container c3b4b0922cf30339f6ce5bfe524cf7702d47af06b7608b0df0a26e785bdfcdc4: Status 404 returned error can't find the container with id c3b4b0922cf30339f6ce5bfe524cf7702d47af06b7608b0df0a26e785bdfcdc4 Nov 03 19:43:42.221022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.220846 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts"] Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.248916 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249020 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249153 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249204 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249230 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249276 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqndq\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249327 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249466 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.249659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.249509 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.252920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.250658 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.252920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.251238 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.252920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.252157 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.254710 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.254180 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.255440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.255394 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.256873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.256338 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.258064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.258026 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.272734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.270911 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.272734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.271584 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqndq\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq\") pod \"e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.356727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.356687 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:42.507905 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:42.507542 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb50808_1204_404f_83c6_3125bddee698.slice/crio-862dbb832cd5e211fb343aa41c2338976aad9219a4d539780530ce000419fff5 WatchSource:0}: Error finding container 862dbb832cd5e211fb343aa41c2338976aad9219a4d539780530ce000419fff5: Status 404 returned error can't find the container with id 862dbb832cd5e211fb343aa41c2338976aad9219a4d539780530ce000419fff5 Nov 03 19:43:42.528314 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.528278 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl"] Nov 03 19:43:42.672930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.672901 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2"] Nov 03 19:43:42.693835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.693802 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:42.693944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.693882 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:42.694063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:42.693947 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:43.019298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.019247 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" event={"ID":"f98a9b0a-1de9-4a96-95b7-82a72f313b19","Type":"ContainerStarted","Data":"c7cd285058234a2d165cca2c7eed52154cc61769d696b1d2bea89ef689182809"} Nov 03 19:43:43.019574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.019543 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" event={"ID":"f98a9b0a-1de9-4a96-95b7-82a72f313b19","Type":"ContainerStarted","Data":"c3b4b0922cf30339f6ce5bfe524cf7702d47af06b7608b0df0a26e785bdfcdc4"} Nov 03 19:43:43.022190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.022164 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" event={"ID":"e3829410-2108-4749-8b62-d32ee82b7e99","Type":"ContainerDied","Data":"9d17410a236077fecaebfc2fa896f0df04c5dfc964348f91fe47727e59639658"} Nov 03 19:43:43.022368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.022354 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d17410a236077fecaebfc2fa896f0df04c5dfc964348f91fe47727e59639658" Nov 03 19:43:43.022589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.022577 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n" Nov 03 19:43:43.057111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.054147 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" event={"ID":"0fb50808-1204-404f-83c6-3125bddee698","Type":"ContainerStarted","Data":"5a41b0acab7ff4b93d2361c084eadc0454d9fcaabe57efff3a6b2aca2ab6ff8d"} Nov 03 19:43:43.057111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.054203 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" event={"ID":"0fb50808-1204-404f-83c6-3125bddee698","Type":"ContainerStarted","Data":"862dbb832cd5e211fb343aa41c2338976aad9219a4d539780530ce000419fff5"} Nov 03 19:43:43.057342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.057313 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" event={"ID":"fb82e649-4d20-4b2f-8457-deec1f0d0d9b","Type":"ContainerStarted","Data":"08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3"} Nov 03 19:43:43.057403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.057366 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" event={"ID":"fb82e649-4d20-4b2f-8457-deec1f0d0d9b","Type":"ContainerStarted","Data":"3fa76012f6757491937e77d1f86e21553675b143b220a75aad63f8bf1dc8212b"} Nov 03 19:43:43.077001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.069442 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" podStartSLOduration=5.069426154 podStartE2EDuration="5.069426154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:43.061545093 +0000 UTC m=+989.402658578" watchObservedRunningTime="2026-11-03 19:43:43.069426154 +0000 UTC m=+989.410539641" Nov 03 19:43:43.118719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.118658 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" podStartSLOduration=4.118634924 podStartE2EDuration="4.118634924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:43.095751586 +0000 UTC m=+989.436865072" watchObservedRunningTime="2026-11-03 19:43:43.118634924 +0000 UTC m=+989.459748412" Nov 03 19:43:43.156763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.156701 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podStartSLOduration=3.156681497 podStartE2EDuration="3.156681497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:43.140301945 +0000 UTC m=+989.481415432" watchObservedRunningTime="2026-11-03 19:43:43.156681497 +0000 UTC m=+989.497794982" Nov 03 19:43:43.357706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.357166 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:43.360328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.360292 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:43.360584 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:43.360557 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:44.122995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.121752 5233 generic.go:356] "Generic (PLEG): container finished" podID="0fb50808-1204-404f-83c6-3125bddee698" containerID="5a41b0acab7ff4b93d2361c084eadc0454d9fcaabe57efff3a6b2aca2ab6ff8d" exitCode=0 Nov 03 19:43:44.122995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.122269 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" event={"ID":"0fb50808-1204-404f-83c6-3125bddee698","Type":"ContainerDied","Data":"5a41b0acab7ff4b93d2361c084eadc0454d9fcaabe57efff3a6b2aca2ab6ff8d"} Nov 03 19:43:44.148003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.127605 5233 generic.go:356] "Generic (PLEG): container finished" podID="f98a9b0a-1de9-4a96-95b7-82a72f313b19" containerID="c7cd285058234a2d165cca2c7eed52154cc61769d696b1d2bea89ef689182809" exitCode=0 Nov 03 19:43:44.148003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.137830 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" event={"ID":"f98a9b0a-1de9-4a96-95b7-82a72f313b19","Type":"ContainerDied","Data":"c7cd285058234a2d165cca2c7eed52154cc61769d696b1d2bea89ef689182809"} Nov 03 19:43:44.357610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.357576 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:44.357769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.357632 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:44.547754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.547690 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4"] Nov 03 19:43:44.553625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.552745 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.555358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.555334 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4"] Nov 03 19:43:44.647524 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.647481 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc882\" (UniqueName: \"kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.647725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.647644 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.647854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.647835 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.735919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.735877 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:43:44.749549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.749519 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.749706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.749690 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.749823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.749807 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc882\" (UniqueName: \"kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.750285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.750261 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.752063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.752044 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.764326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.764303 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc882\" (UniqueName: \"kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882\") pod \"maas-api-revoke-sub-1d736b56-dfks4\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:44.870030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:44.869922 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:45.353154 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:45.353089 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbced5b_4b7e_496d_9461_dd99fe6aadd5.slice/crio-9e15bc73798184d82a587cfba2d8f7e10b4305a9683183cdec9822d629a03d4a WatchSource:0}: Error finding container 9e15bc73798184d82a587cfba2d8f7e10b4305a9683183cdec9822d629a03d4a: Status 404 returned error can't find the container with id 9e15bc73798184d82a587cfba2d8f7e10b4305a9683183cdec9822d629a03d4a Nov 03 19:43:45.357179 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.357151 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:45.357368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.357342 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4"] Nov 03 19:43:45.357480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.357451 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:45.528767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.528737 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:45.570000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.569952 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm\" (UniqueName: \"kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm\") pod \"0fb50808-1204-404f-83c6-3125bddee698\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " Nov 03 19:43:45.570262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.570202 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca\") pod \"0fb50808-1204-404f-83c6-3125bddee698\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " Nov 03 19:43:45.570262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.570261 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca\") pod \"0fb50808-1204-404f-83c6-3125bddee698\" (UID: \"0fb50808-1204-404f-83c6-3125bddee698\") " Nov 03 19:43:45.572672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.572640 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca" pod "0fb50808-1204-404f-83c6-3125bddee698" (UID: "0fb50808-1204-404f-83c6-3125bddee698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:45.624587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.624544 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca" pod "0fb50808-1204-404f-83c6-3125bddee698" (UID: "0fb50808-1204-404f-83c6-3125bddee698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:45.624762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.624738 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm" pod "0fb50808-1204-404f-83c6-3125bddee698" (UID: "0fb50808-1204-404f-83c6-3125bddee698"). InnerVolumeSpecName "kube-api-access-xknwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:45.633785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.633753 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:45.673040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.673013 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca\") pod \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " Nov 03 19:43:45.673166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.673086 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q\" (UniqueName: \"kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q\") pod \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " Nov 03 19:43:45.673166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.673125 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca\") pod \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\" (UID: \"f98a9b0a-1de9-4a96-95b7-82a72f313b19\") " Nov 03 19:43:45.673546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.673532 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xknwm\" (UniqueName: \"kubernetes.io/projected/0fb50808-1204-404f-83c6-3125bddee698-kube-api-access-xknwm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:45.673583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.673548 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb50808-1204-404f-83c6-3125bddee698-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:45.673583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.673558 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb50808-1204-404f-83c6-3125bddee698-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:45.674098 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.674071 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca" pod "f98a9b0a-1de9-4a96-95b7-82a72f313b19" (UID: "f98a9b0a-1de9-4a96-95b7-82a72f313b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:45.675570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.675546 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca" pod "f98a9b0a-1de9-4a96-95b7-82a72f313b19" (UID: "f98a9b0a-1de9-4a96-95b7-82a72f313b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:45.676640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.676620 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q" pod "f98a9b0a-1de9-4a96-95b7-82a72f313b19" (UID: "f98a9b0a-1de9-4a96-95b7-82a72f313b19"). InnerVolumeSpecName "kube-api-access-lsl4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:45.775586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.775546 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:45.775689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.775589 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl4q\" (UniqueName: \"kubernetes.io/projected/f98a9b0a-1de9-4a96-95b7-82a72f313b19-kube-api-access-lsl4q\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:45.775689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:45.775600 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98a9b0a-1de9-4a96-95b7-82a72f313b19-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:46.159138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.159046 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" Nov 03 19:43:46.159138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.159085 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl" event={"ID":"0fb50808-1204-404f-83c6-3125bddee698","Type":"ContainerDied","Data":"862dbb832cd5e211fb343aa41c2338976aad9219a4d539780530ce000419fff5"} Nov 03 19:43:46.159138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.159133 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862dbb832cd5e211fb343aa41c2338976aad9219a4d539780530ce000419fff5" Nov 03 19:43:46.160895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.160877 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" Nov 03 19:43:46.160895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.160890 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts" event={"ID":"f98a9b0a-1de9-4a96-95b7-82a72f313b19","Type":"ContainerDied","Data":"c3b4b0922cf30339f6ce5bfe524cf7702d47af06b7608b0df0a26e785bdfcdc4"} Nov 03 19:43:46.161099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.160930 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3b4b0922cf30339f6ce5bfe524cf7702d47af06b7608b0df0a26e785bdfcdc4" Nov 03 19:43:46.163236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.163213 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" event={"ID":"6bbced5b-4b7e-496d-9461-dd99fe6aadd5","Type":"ContainerStarted","Data":"e0ea5b971b5706bc04f72cfa8f0ff1a5dc35c42544b03d60f4ad0437e4d16011"} Nov 03 19:43:46.163327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.163258 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" event={"ID":"6bbced5b-4b7e-496d-9461-dd99fe6aadd5","Type":"ContainerStarted","Data":"9e15bc73798184d82a587cfba2d8f7e10b4305a9683183cdec9822d629a03d4a"} Nov 03 19:43:46.239184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.239135 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" podStartSLOduration=2.23911778 podStartE2EDuration="2.23911778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:46.184431016 +0000 UTC m=+992.525544514" watchObservedRunningTime="2026-11-03 19:43:46.23911778 +0000 UTC m=+992.580231283" Nov 03 19:43:46.357632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.357287 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:46.357632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:46.357335 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:47.263962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:47.263926 5233 generic.go:356] "Generic (PLEG): container finished" podID="6bbced5b-4b7e-496d-9461-dd99fe6aadd5" containerID="e0ea5b971b5706bc04f72cfa8f0ff1a5dc35c42544b03d60f4ad0437e4d16011" exitCode=0 Nov 03 19:43:47.264394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:47.264365 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" event={"ID":"6bbced5b-4b7e-496d-9461-dd99fe6aadd5","Type":"ContainerDied","Data":"e0ea5b971b5706bc04f72cfa8f0ff1a5dc35c42544b03d60f4ad0437e4d16011"} Nov 03 19:43:47.360413 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:47.360377 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:47.360951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:47.360922 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:48.357924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.357890 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:48.358093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.357955 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:48.544409 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.544338 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:48.637482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.637448 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca\") pod \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " Nov 03 19:43:48.637808 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.637777 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca\") pod \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " Nov 03 19:43:48.637808 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.637793 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca" pod "6bbced5b-4b7e-496d-9461-dd99fe6aadd5" (UID: "6bbced5b-4b7e-496d-9461-dd99fe6aadd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:48.637954 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.637877 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882\" (UniqueName: \"kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882\") pod \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\" (UID: \"6bbced5b-4b7e-496d-9461-dd99fe6aadd5\") " Nov 03 19:43:48.639258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.639237 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:48.640501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.640474 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca" pod "6bbced5b-4b7e-496d-9461-dd99fe6aadd5" (UID: "6bbced5b-4b7e-496d-9461-dd99fe6aadd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:48.640597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.640565 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882" pod "6bbced5b-4b7e-496d-9461-dd99fe6aadd5" (UID: "6bbced5b-4b7e-496d-9461-dd99fe6aadd5"). InnerVolumeSpecName "kube-api-access-wc882". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:48.753815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.753737 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:48.754087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:48.754065 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc882\" (UniqueName: \"kubernetes.io/projected/6bbced5b-4b7e-496d-9461-dd99fe6aadd5-kube-api-access-wc882\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:49.289318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:49.289285 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" event={"ID":"6bbced5b-4b7e-496d-9461-dd99fe6aadd5","Type":"ContainerDied","Data":"9e15bc73798184d82a587cfba2d8f7e10b4305a9683183cdec9822d629a03d4a"} Nov 03 19:43:49.289318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:49.289302 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4" Nov 03 19:43:49.289522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:49.289337 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e15bc73798184d82a587cfba2d8f7e10b4305a9683183cdec9822d629a03d4a" Nov 03 19:43:49.357386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:49.357352 5233 patch_prober.go:29] interesting pod/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:43:49.357555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:49.357406 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 19:43:50.714017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:50.706737 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:50.714017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:50.707295 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:50.714017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:50.708281 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:43:51.289640 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:43:51.289607 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:43:51.534512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.533687 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4"] Nov 03 19:43:51.534895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.534867 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f98a9b0a-1de9-4a96-95b7-82a72f313b19" containerName="revoke-keys" Nov 03 19:43:51.534895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.534895 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98a9b0a-1de9-4a96-95b7-82a72f313b19" containerName="revoke-keys" Nov 03 19:43:51.535160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.534920 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6bbced5b-4b7e-496d-9461-dd99fe6aadd5" containerName="revoke-keys" Nov 03 19:43:51.535160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.534928 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbced5b-4b7e-496d-9461-dd99fe6aadd5" containerName="revoke-keys" Nov 03 19:43:51.535160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.534950 5233 cpu_manager.go:414] "containerMap: removing container" podUID="0fb50808-1204-404f-83c6-3125bddee698" containerName="revoke-keys" Nov 03 19:43:51.535160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.534958 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb50808-1204-404f-83c6-3125bddee698" containerName="revoke-keys" Nov 03 19:43:51.535375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.535166 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbced5b-4b7e-496d-9461-dd99fe6aadd5" containerName="revoke-keys" Nov 03 19:43:51.535375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.535180 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb50808-1204-404f-83c6-3125bddee698" containerName="revoke-keys" Nov 03 19:43:51.535375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.535205 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98a9b0a-1de9-4a96-95b7-82a72f313b19" containerName="revoke-keys" Nov 03 19:43:51.544401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.539827 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.544401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.543112 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4"] Nov 03 19:43:51.617916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.617884 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.618136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.618085 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.618136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.618115 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.618257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.618160 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.618321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.618278 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.618427 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.618404 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.618489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.618476 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzdw\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.720450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720414 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzdw\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720526 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720688 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720716 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720767 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720806 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.720921 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.721839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.721810 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.723734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.723675 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.724061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.724034 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.724193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.724169 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.729417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.729392 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.737254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.737229 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzdw\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.737355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.737341 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8cp4\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:51.868703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:51.868615 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:52.190541 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:52.190504 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c59966_294c_4f89_9d00_d6077ec55c35.slice/crio-7478a98367cafd651a82dafc734542ece42afdac00b8bf96adb108cecb0ffdca WatchSource:0}: Error finding container 7478a98367cafd651a82dafc734542ece42afdac00b8bf96adb108cecb0ffdca: Status 404 returned error can't find the container with id 7478a98367cafd651a82dafc734542ece42afdac00b8bf96adb108cecb0ffdca Nov 03 19:43:52.190837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.190813 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4"] Nov 03 19:43:52.193808 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.193773 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:52.193912 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.193843 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:43:52.344247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.344192 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" event={"ID":"a0c59966-294c-4f89-9d00-d6077ec55c35","Type":"ContainerStarted","Data":"b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55"} Nov 03 19:43:52.344457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.344292 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" event={"ID":"a0c59966-294c-4f89-9d00-d6077ec55c35","Type":"ContainerStarted","Data":"7478a98367cafd651a82dafc734542ece42afdac00b8bf96adb108cecb0ffdca"} Nov 03 19:43:52.344457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.344317 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:52.967365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.967296 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" podStartSLOduration=1.9672734200000002 podStartE2EDuration="1.96727342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:52.380305512 +0000 UTC m=+998.721418997" watchObservedRunningTime="2026-11-03 19:43:52.96727342 +0000 UTC m=+999.308386906" Nov 03 19:43:52.969153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.969120 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr"] Nov 03 19:43:52.975929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.975907 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:52.990465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.990443 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:43:52.990729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.990711 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:43:52.994113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:52.994089 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr"] Nov 03 19:43:53.069357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.069291 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxk5q\" (UniqueName: \"kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.069687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.069649 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.069890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.069870 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.172264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.172201 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxk5q\" (UniqueName: \"kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.172465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.172326 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.172465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.172386 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.173279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.173250 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.175275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.175251 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.198016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.186133 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxk5q\" (UniqueName: \"kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q\") pod \"maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.316488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.316394 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:53.360412 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.360381 5233 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-q8cp4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:43:53.360594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.360436 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" podUID="a0c59966-294c-4f89-9d00-d6077ec55c35" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:43:53.615466 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:53.615435 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbe5da3_9273_44ac_9b7b_1f2bfa41186d.slice/crio-c016bf102db099bccd48bc29d29179c823b2b0351e7d6f62857b6f722d1141d6 WatchSource:0}: Error finding container c016bf102db099bccd48bc29d29179c823b2b0351e7d6f62857b6f722d1141d6: Status 404 returned error can't find the container with id c016bf102db099bccd48bc29d29179c823b2b0351e7d6f62857b6f722d1141d6 Nov 03 19:43:53.618773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:53.618750 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr"] Nov 03 19:43:54.368659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:54.368565 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" event={"ID":"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d","Type":"ContainerStarted","Data":"790a1450a4eea90713b7053e10170f474e3673755a1073b8b2f862c234cdc017"} Nov 03 19:43:54.368659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:54.368621 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" event={"ID":"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d","Type":"ContainerStarted","Data":"c016bf102db099bccd48bc29d29179c823b2b0351e7d6f62857b6f722d1141d6"} Nov 03 19:43:54.396119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:54.396069 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" podStartSLOduration=2.396055315 podStartE2EDuration="2.396055315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:54.387565985 +0000 UTC m=+1000.728679472" watchObservedRunningTime="2026-11-03 19:43:54.396055315 +0000 UTC m=+1000.737168846" Nov 03 19:43:55.382853 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:55.382823 5233 generic.go:356] "Generic (PLEG): container finished" podID="1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" containerID="790a1450a4eea90713b7053e10170f474e3673755a1073b8b2f862c234cdc017" exitCode=0 Nov 03 19:43:55.383264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:55.382912 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" event={"ID":"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d","Type":"ContainerDied","Data":"790a1450a4eea90713b7053e10170f474e3673755a1073b8b2f862c234cdc017"} Nov 03 19:43:56.358537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.358504 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:43:56.666905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.666875 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:56.742798 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.742762 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q\" (UniqueName: \"kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q\") pod \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " Nov 03 19:43:56.743003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.742985 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca\") pod \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " Nov 03 19:43:56.743094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.743037 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca\") pod \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\" (UID: \"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d\") " Nov 03 19:43:56.743353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.743323 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca" pod "1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" (UID: "1bbe5da3-9273-44ac-9b7b-1f2bfa41186d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:43:56.743802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.743771 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:56.745152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.745129 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q" pod "1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" (UID: "1bbe5da3-9273-44ac-9b7b-1f2bfa41186d"). InnerVolumeSpecName "kube-api-access-sxk5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:43:56.745278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.745225 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca" pod "1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" (UID: "1bbe5da3-9273-44ac-9b7b-1f2bfa41186d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:43:56.794240 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.793381 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn"] Nov 03 19:43:56.794637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.794614 5233 cpu_manager.go:414] "containerMap: removing container" podUID="1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" containerName="revoke-keys" Nov 03 19:43:56.794743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.794639 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" containerName="revoke-keys" Nov 03 19:43:56.794912 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.794886 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" containerName="revoke-keys" Nov 03 19:43:56.800327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.800162 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:56.802933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.802225 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn"] Nov 03 19:43:56.819494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.819455 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-28ef3ba9\"" Nov 03 19:43:56.845326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.845295 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:56.845326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.845325 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxk5q\" (UniqueName: \"kubernetes.io/projected/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d-kube-api-access-sxk5q\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:43:56.923805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.923180 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-28ef3ba9-dockercfg-vjjjq\"" Nov 03 19:43:56.957016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.950001 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7b6\" (UniqueName: \"kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6\") pod \"payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:56.957016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:56.950171 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume\") pod \"payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:57.100096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.093101 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume\") pod \"payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:57.100096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.093489 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7b6\" (UniqueName: \"kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6\") pod \"payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:57.100096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.094888 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume\") pod \"payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:57.123772 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.123734 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7b6\" (UniqueName: \"kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6\") pod \"payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:57.124617 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.124591 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:57.411515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.411467 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" event={"ID":"1bbe5da3-9273-44ac-9b7b-1f2bfa41186d","Type":"ContainerDied","Data":"c016bf102db099bccd48bc29d29179c823b2b0351e7d6f62857b6f722d1141d6"} Nov 03 19:43:57.411720 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.411529 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c016bf102db099bccd48bc29d29179c823b2b0351e7d6f62857b6f722d1141d6" Nov 03 19:43:57.411720 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.411596 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr" Nov 03 19:43:57.530203 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:43:57.529099 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1b9326_e198_47dd_bb52_f067f4173d74.slice/crio-628a66248c5bf2a5537c5b668bdda16c58e0c07b589f63ccf84669e8c166825c WatchSource:0}: Error finding container 628a66248c5bf2a5537c5b668bdda16c58e0c07b589f63ccf84669e8c166825c: Status 404 returned error can't find the container with id 628a66248c5bf2a5537c5b668bdda16c58e0c07b589f63ccf84669e8c166825c Nov 03 19:43:57.559524 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:57.559495 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn"] Nov 03 19:43:58.427125 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:58.427093 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" event={"ID":"8d1b9326-e198-47dd-bb52-f067f4173d74","Type":"ContainerStarted","Data":"006874786b4105f60af4e1a8ebc84ce7f648642d4d7d3256efe61780510eaec1"} Nov 03 19:43:58.427522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:58.427172 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" event={"ID":"8d1b9326-e198-47dd-bb52-f067f4173d74","Type":"ContainerStarted","Data":"628a66248c5bf2a5537c5b668bdda16c58e0c07b589f63ccf84669e8c166825c"} Nov 03 19:43:58.427522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:58.427248 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:43:58.457546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:43:58.457498 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" podStartSLOduration=2.457484418 podStartE2EDuration="2.457484418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:58.450291224 +0000 UTC m=+1004.791404713" watchObservedRunningTime="2026-11-03 19:43:58.457484418 +0000 UTC m=+1004.798597902" Nov 03 19:44:02.116052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.116013 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb"] Nov 03 19:44:02.121704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.121680 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.131873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.131845 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9b7a388c-dockercfg-qmlm8\"" Nov 03 19:44:02.140634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.140602 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb"] Nov 03 19:44:02.148613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.148181 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9b7a388c\"" Nov 03 19:44:02.168269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.168237 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.168570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.168546 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85649\" (UniqueName: \"kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649\") pod \"payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.445055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.439508 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.445055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.439795 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85649\" (UniqueName: \"kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649\") pod \"payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.445055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.441132 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.577938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.577604 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j"] Nov 03 19:44:02.577938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.577879 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85649\" (UniqueName: \"kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649\") pod \"payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.583876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.583853 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.585595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.585564 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j"] Nov 03 19:44:02.587987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.587951 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:44:02.588338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.588311 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:44:02.651656 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.651616 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.651848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.651687 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqtp\" (UniqueName: \"kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.651848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.651780 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.752804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.752769 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.752992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.752841 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqtp\" (UniqueName: \"kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.752992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.752909 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.753563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.753543 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.755166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.755135 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.757113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.757097 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:02.765474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.765453 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqtp\" (UniqueName: \"kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp\") pod \"maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:02.911280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:02.911249 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:03.053719 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:44:03.053642 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db8eeca_6fed_490e_9f00_c859998d4876.slice/crio-7b8e6eec3daa93d3a4a323221d9bdf19fef4f63f653ce7b12edc845c844f0dde WatchSource:0}: Error finding container 7b8e6eec3daa93d3a4a323221d9bdf19fef4f63f653ce7b12edc845c844f0dde: Status 404 returned error can't find the container with id 7b8e6eec3daa93d3a4a323221d9bdf19fef4f63f653ce7b12edc845c844f0dde Nov 03 19:44:03.056256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.056233 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb"] Nov 03 19:44:03.222037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.221245 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw"] Nov 03 19:44:03.234191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.234159 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47"] Nov 03 19:44:03.253828 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:44:03.252946 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396d583b_6593_4eff_9eee_7d74a9f4934f.slice/crio-e05d2eeea264259cf9aa1df8f1f7b884fc0e4b9cb605cc7506c646a4cee8f34c WatchSource:0}: Error finding container e05d2eeea264259cf9aa1df8f1f7b884fc0e4b9cb605cc7506c646a4cee8f34c: Status 404 returned error can't find the container with id e05d2eeea264259cf9aa1df8f1f7b884fc0e4b9cb605cc7506c646a4cee8f34c Nov 03 19:44:03.264281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.264211 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j"] Nov 03 19:44:03.694714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.694598 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" event={"ID":"396d583b-6593-4eff-9eee-7d74a9f4934f","Type":"ContainerStarted","Data":"e05d2eeea264259cf9aa1df8f1f7b884fc0e4b9cb605cc7506c646a4cee8f34c"} Nov 03 19:44:03.717027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.709417 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" event={"ID":"9db8eeca-6fed-490e-9f00-c859998d4876","Type":"ContainerStarted","Data":"f13ed035c6c84020fe6eb774a74e8e348ca0b60bc08b54f0eaf54bfa0d0dc76b"} Nov 03 19:44:03.717027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.709530 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" event={"ID":"9db8eeca-6fed-490e-9f00-c859998d4876","Type":"ContainerStarted","Data":"7b8e6eec3daa93d3a4a323221d9bdf19fef4f63f653ce7b12edc845c844f0dde"} Nov 03 19:44:03.717027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.710030 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:03.745013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:03.733345 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" podStartSLOduration=1.733324978 podStartE2EDuration="1.733324978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:03.732378489 +0000 UTC m=+1010.073491975" watchObservedRunningTime="2026-11-03 19:44:03.733324978 +0000 UTC m=+1010.074438463" Nov 03 19:44:04.721180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:04.721144 5233 generic.go:356] "Generic (PLEG): container finished" podID="396d583b-6593-4eff-9eee-7d74a9f4934f" containerID="c87a718f05802219d91345f77b722f5a60b89f8df11640249c19c433fa9e3ae5" exitCode=0 Nov 03 19:44:04.721652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:04.721229 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" event={"ID":"396d583b-6593-4eff-9eee-7d74a9f4934f","Type":"ContainerDied","Data":"c87a718f05802219d91345f77b722f5a60b89f8df11640249c19c433fa9e3ae5"} Nov 03 19:44:05.907384 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:05.907355 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:06.006058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.004810 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp\" (UniqueName: \"kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp\") pod \"396d583b-6593-4eff-9eee-7d74a9f4934f\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " Nov 03 19:44:06.006058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.005068 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca\") pod \"396d583b-6593-4eff-9eee-7d74a9f4934f\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " Nov 03 19:44:06.006058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.005326 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca\") pod \"396d583b-6593-4eff-9eee-7d74a9f4934f\" (UID: \"396d583b-6593-4eff-9eee-7d74a9f4934f\") " Nov 03 19:44:06.011698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.011337 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca" pod "396d583b-6593-4eff-9eee-7d74a9f4934f" (UID: "396d583b-6593-4eff-9eee-7d74a9f4934f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:06.058105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.057875 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca" pod "396d583b-6593-4eff-9eee-7d74a9f4934f" (UID: "396d583b-6593-4eff-9eee-7d74a9f4934f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:06.067529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.067304 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp" pod "396d583b-6593-4eff-9eee-7d74a9f4934f" (UID: "396d583b-6593-4eff-9eee-7d74a9f4934f"). InnerVolumeSpecName "kube-api-access-9kqtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:06.107308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.107277 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:06.107440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.107309 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqtp\" (UniqueName: \"kubernetes.io/projected/396d583b-6593-4eff-9eee-7d74a9f4934f-kube-api-access-9kqtp\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:06.107440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.107327 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d583b-6593-4eff-9eee-7d74a9f4934f-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:06.745317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.745288 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" Nov 03 19:44:06.745495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.745318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j" event={"ID":"396d583b-6593-4eff-9eee-7d74a9f4934f","Type":"ContainerDied","Data":"e05d2eeea264259cf9aa1df8f1f7b884fc0e4b9cb605cc7506c646a4cee8f34c"} Nov 03 19:44:06.745495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:06.745369 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05d2eeea264259cf9aa1df8f1f7b884fc0e4b9cb605cc7506c646a4cee8f34c" Nov 03 19:44:08.221597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.221527 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" podUID="a8236266-058e-429d-a8b4-89e376265d5a" containerName="payload-processing" containerID="cri-o://0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e" gracePeriod=25 Nov 03 19:44:08.234852 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.234793 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" podUID="1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" containerName="payload-pre-processing" containerID="cri-o://ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d" gracePeriod=25 Nov 03 19:44:08.675991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.675952 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:44:08.726625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.726595 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:44:08.758164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.758118 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7\" (UniqueName: \"kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7\") pod \"a8236266-058e-429d-a8b4-89e376265d5a\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " Nov 03 19:44:08.758399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.758377 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume\") pod \"a8236266-058e-429d-a8b4-89e376265d5a\" (UID: \"a8236266-058e-429d-a8b4-89e376265d5a\") " Nov 03 19:44:08.759474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.759444 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume" pod "a8236266-058e-429d-a8b4-89e376265d5a" (UID: "a8236266-058e-429d-a8b4-89e376265d5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:08.761798 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.761770 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7" pod "a8236266-058e-429d-a8b4-89e376265d5a" (UID: "a8236266-058e-429d-a8b4-89e376265d5a"). InnerVolumeSpecName "kube-api-access-nk6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:08.771495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.771467 5233 generic.go:356] "Generic (PLEG): container finished" podID="a8236266-058e-429d-a8b4-89e376265d5a" containerID="0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e" exitCode=0 Nov 03 19:44:08.771604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.771511 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" Nov 03 19:44:08.771604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.771556 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" event={"ID":"a8236266-058e-429d-a8b4-89e376265d5a","Type":"ContainerDied","Data":"0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e"} Nov 03 19:44:08.771728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.771638 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw" event={"ID":"a8236266-058e-429d-a8b4-89e376265d5a","Type":"ContainerDied","Data":"2c2e6a5c2da0990152f713deec47a4a08a87378f40c85708a80c0e1653a80be1"} Nov 03 19:44:08.771728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.771671 5233 scope.go:122] "RemoveContainer" containerID="0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e" Nov 03 19:44:08.778038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.777938 5233 generic.go:356] "Generic (PLEG): container finished" podID="1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" containerID="ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d" exitCode=0 Nov 03 19:44:08.778134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.778036 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" Nov 03 19:44:08.778224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.778200 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" event={"ID":"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3","Type":"ContainerDied","Data":"ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d"} Nov 03 19:44:08.778297 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.778267 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47" event={"ID":"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3","Type":"ContainerDied","Data":"adbddbec0f1c105bca956b08ff99b4238f79aa7a6cf1e2a14718477e79dcaa37"} Nov 03 19:44:08.814994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.814252 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw"] Nov 03 19:44:08.819635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.818180 5233 scope.go:122] "RemoveContainer" containerID="0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e" Nov 03 19:44:08.819635 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:08.818577 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e\": container with ID starting with 0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e not found: ID does not exist" containerID="0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e" Nov 03 19:44:08.819635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.818607 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e"} err="failed to get container status \"0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e\": rpc error: code = NotFound desc = could not find container \"0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e\": container with ID starting with 0d1b6e02c6f791cd7609bbb1bab8d7f7ac8a8fdffe696fe2cb9f25fa1da7a98e not found: ID does not exist" Nov 03 19:44:08.819635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.818633 5233 scope.go:122] "RemoveContainer" containerID="ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d" Nov 03 19:44:08.825510 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.825316 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca12e9fb-58cf74f74c-m6mpw"] Nov 03 19:44:08.864557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.863193 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume\") pod \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " Nov 03 19:44:08.864557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.863350 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n\" (UniqueName: \"kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n\") pod \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\" (UID: \"1b9c03c1-f5c6-4b0c-b59b-96927570e2a3\") " Nov 03 19:44:08.864557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.864086 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6k7\" (UniqueName: \"kubernetes.io/projected/a8236266-058e-429d-a8b4-89e376265d5a-kube-api-access-nk6k7\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:08.864557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.864109 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8236266-058e-429d-a8b4-89e376265d5a-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:08.865502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.865471 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume" pod "1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" (UID: "1b9c03c1-f5c6-4b0c-b59b-96927570e2a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:08.867016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.866916 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n" pod "1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" (UID: "1b9c03c1-f5c6-4b0c-b59b-96927570e2a3"). InnerVolumeSpecName "kube-api-access-d728n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:08.876611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.876585 5233 scope.go:122] "RemoveContainer" containerID="ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d" Nov 03 19:44:08.877020 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:08.876961 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d\": container with ID starting with ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d not found: ID does not exist" containerID="ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d" Nov 03 19:44:08.877116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.877030 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d"} err="failed to get container status \"ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d\": rpc error: code = NotFound desc = could not find container \"ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d\": container with ID starting with ecbd16622baf766d072e75bdffba4f34a7233df3366f8fbf2b70e549e63d532d not found: ID does not exist" Nov 03 19:44:08.966121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.966087 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:08.966121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:08.966116 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d728n\" (UniqueName: \"kubernetes.io/projected/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3-kube-api-access-d728n\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:09.114033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:09.114000 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47"] Nov 03 19:44:09.121560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:09.121489 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca12e9fb-546c8645c9-zcl47"] Nov 03 19:44:09.443084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:09.442991 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:44:09.845039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:09.844318 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" path="/var/lib/kubelet/pods/1b9c03c1-f5c6-4b0c-b59b-96927570e2a3/volumes" Nov 03 19:44:09.845039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:09.844940 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8236266-058e-429d-a8b4-89e376265d5a" path="/var/lib/kubelet/pods/a8236266-058e-429d-a8b4-89e376265d5a/volumes" Nov 03 19:44:12.530990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:12.528511 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:12.672397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:12.672341 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:12.688000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:12.676422 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:44:12.939099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:12.879747 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:13.011001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.008330 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:44:13.294070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.287797 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:44:13.336454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.336403 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:13.609054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.596639 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:13.634662 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.634611 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:44:13.734567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.727352 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:13.734567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.731366 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:44:13.774260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.771755 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:13.774260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.772180 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:44:13.774260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.773594 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:44:13.774260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:13.773656 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:44:14.378997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:14.353633 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:44:14.756040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:14.740571 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:14.801003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:14.789867 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:44:14.844324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:14.844283 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:44:14.890999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:14.880484 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:44:14.990993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:14.988318 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:15.059048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:15.019895 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:44:15.422049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:15.389775 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:15.534341 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:15.155715 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:15.607998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:15.155750 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:44:15.931073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:15.931021 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:44:16.075501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.001200 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:44:16.276081 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.247134 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded" Nov 03 19:44:16.299847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.299796 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:16.368995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.365694 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:44:16.594013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.575506 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:16.846470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.846370 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:44:16.939411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:16.939358 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:44:16.993238 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:16.993178 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.087s" Nov 03 19:44:17.101466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.099405 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj"] Nov 03 19:44:17.101466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.099530 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:44:17.101466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.099555 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5"] Nov 03 19:44:17.153993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.148535 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:44:17.266994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.050888 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2n225" podUID="784da5d0-b4e7-4590-a365-2be891af94c2" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 03 19:44:17.266994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.075548 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:17.345185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.345139 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:17.356509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.075605 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:44:17.591161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.591104 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:17.647082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.644172 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:44:17.839300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.839267 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:44:17.839572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.839557 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:44:17.839708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.839697 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:44:17.839876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.839865 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:44:18.158015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:17.982292 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:18.363252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:18.287193 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:44:18.363252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:18.359168 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:44:18.366709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:18.365905 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:18.476338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:18.476284 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 19:44:18.708763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:18.708718 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:44:18.950300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:18.950190 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:19.024998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.005471 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:19.140993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.061890 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:44:19.675123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.279105 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:19.679049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.410543 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:44:19.679213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.517344 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:44:19.680944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.517379 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:19.680944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.606757 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:44:19.725661 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:19.712146 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.872s" Nov 03 19:44:19.725661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.712378 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:19.725661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.712452 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:44:19.725661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.712505 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:44:19.725661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.716550 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:19.964125 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.963207 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:44:19.964125 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.963272 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:44:19.986037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.965281 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:44:19.986336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.986312 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:44:19.986528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.986513 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:44:20.011994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:19.986688 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:44:20.057069 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:20.057032 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr"] Nov 03 19:44:20.148047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:20.069447 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ca12e9fb-rfgbr"] Nov 03 19:44:20.148047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:20.080959 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst"] Nov 03 19:44:20.148047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:20.081196 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" podUID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" containerName="istio-proxy" containerID="cri-o://fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a" gracePeriod=30 Nov 03 19:44:21.541994 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:21.540007 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:44:21.541994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:21.540218 5233 patch_prober.go:29] interesting pod/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:44:21.541994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:21.540264 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" podUID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:44:21.855220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:21.855131 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbe5da3-9273-44ac-9b7b-1f2bfa41186d" path="/var/lib/kubelet/pods/1bbe5da3-9273-44ac-9b7b-1f2bfa41186d/volumes" Nov 03 19:44:22.240089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.240040 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" podUID="64e3fd4e-4626-44ab-9406-c2e7c9c92480" containerName="payload-processing" containerID="cri-o://6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12" gracePeriod=25 Nov 03 19:44:22.360132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.360076 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" podUID="cf8b0cec-42ec-4bb2-83b2-66092f821b2a" containerName="payload-pre-processing" containerID="cri-o://0bf6bfac97d6a32db0fd54732b1fbf29fe982d35817b8036e3e3e1475b9c4b4e" gracePeriod=25 Nov 03 19:44:22.797520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.797076 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:44:22.872989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.872295 5233 generic.go:356] "Generic (PLEG): container finished" podID="64e3fd4e-4626-44ab-9406-c2e7c9c92480" containerID="6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12" exitCode=0 Nov 03 19:44:22.872989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.872853 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" event={"ID":"64e3fd4e-4626-44ab-9406-c2e7c9c92480","Type":"ContainerDied","Data":"6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12"} Nov 03 19:44:22.872989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.872942 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" event={"ID":"64e3fd4e-4626-44ab-9406-c2e7c9c92480","Type":"ContainerDied","Data":"9888acf4a754ef27e838ccb819cbd5de3f910fa89e0625c5796b54da8a9abeaf"} Nov 03 19:44:22.873273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.873011 5233 scope.go:122] "RemoveContainer" containerID="6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12" Nov 03 19:44:22.873331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.873268 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj" Nov 03 19:44:22.876855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.876826 5233 generic.go:356] "Generic (PLEG): container finished" podID="cf8b0cec-42ec-4bb2-83b2-66092f821b2a" containerID="0bf6bfac97d6a32db0fd54732b1fbf29fe982d35817b8036e3e3e1475b9c4b4e" exitCode=0 Nov 03 19:44:22.876996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.876921 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" event={"ID":"cf8b0cec-42ec-4bb2-83b2-66092f821b2a","Type":"ContainerDied","Data":"0bf6bfac97d6a32db0fd54732b1fbf29fe982d35817b8036e3e3e1475b9c4b4e"} Nov 03 19:44:22.930123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.926812 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume\") pod \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " Nov 03 19:44:22.930123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.926885 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9\" (UniqueName: \"kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9\") pod \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\" (UID: \"64e3fd4e-4626-44ab-9406-c2e7c9c92480\") " Nov 03 19:44:22.936320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.936280 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume" pod "64e3fd4e-4626-44ab-9406-c2e7c9c92480" (UID: "64e3fd4e-4626-44ab-9406-c2e7c9c92480"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:22.939178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.939149 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9" pod "64e3fd4e-4626-44ab-9406-c2e7c9c92480" (UID: "64e3fd4e-4626-44ab-9406-c2e7c9c92480"). InnerVolumeSpecName "kube-api-access-zmwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:22.939512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.939494 5233 scope.go:122] "RemoveContainer" containerID="6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12" Nov 03 19:44:22.939881 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:22.939848 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12\": container with ID starting with 6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12 not found: ID does not exist" containerID="6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12" Nov 03 19:44:22.940027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.939891 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12"} err="failed to get container status \"6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12\": rpc error: code = NotFound desc = could not find container \"6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12\": container with ID starting with 6ceb4a491071695a0b92f7785cae38f7486b94c3ab2e5389e1d21103130d1f12 not found: ID does not exist" Nov 03 19:44:22.968951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:22.968927 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.028544 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume\") pod \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.028791 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl\" (UniqueName: \"kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl\") pod \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\" (UID: \"cf8b0cec-42ec-4bb2-83b2-66092f821b2a\") " Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.029423 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume" pod "cf8b0cec-42ec-4bb2-83b2-66092f821b2a" (UID: "cf8b0cec-42ec-4bb2-83b2-66092f821b2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.029792 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e3fd4e-4626-44ab-9406-c2e7c9c92480-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.029814 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwm9\" (UniqueName: \"kubernetes.io/projected/64e3fd4e-4626-44ab-9406-c2e7c9c92480-kube-api-access-zmwm9\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.029826 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:23.035878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.032857 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl" pod "cf8b0cec-42ec-4bb2-83b2-66092f821b2a" (UID: "cf8b0cec-42ec-4bb2-83b2-66092f821b2a"). InnerVolumeSpecName "kube-api-access-lvxdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:23.149033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.147871 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvxdl\" (UniqueName: \"kubernetes.io/projected/cf8b0cec-42ec-4bb2-83b2-66092f821b2a-kube-api-access-lvxdl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:23.246349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.246317 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj"] Nov 03 19:44:23.257249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.257219 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc935cf-f6c789b6c-w2whj"] Nov 03 19:44:23.841771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.841737 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e3fd4e-4626-44ab-9406-c2e7c9c92480" path="/var/lib/kubelet/pods/64e3fd4e-4626-44ab-9406-c2e7c9c92480/volumes" Nov 03 19:44:23.900022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.892252 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" Nov 03 19:44:23.900022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.892261 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5" event={"ID":"cf8b0cec-42ec-4bb2-83b2-66092f821b2a","Type":"ContainerDied","Data":"e89053f4e50921ac8aa52e09522b459a03373c36a041766ead415865fb6efd0b"} Nov 03 19:44:23.900022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.892350 5233 scope.go:122] "RemoveContainer" containerID="0bf6bfac97d6a32db0fd54732b1fbf29fe982d35817b8036e3e3e1475b9c4b4e" Nov 03 19:44:23.964784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.964742 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5"] Nov 03 19:44:23.971821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:23.971789 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc935cf-8876d59ff-j6xx5"] Nov 03 19:44:25.840447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:25.840409 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8b0cec-42ec-4bb2-83b2-66092f821b2a" path="/var/lib/kubelet/pods/cf8b0cec-42ec-4bb2-83b2-66092f821b2a/volumes" Nov 03 19:44:26.000881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.000845 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8"] Nov 03 19:44:26.002007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.001985 5233 cpu_manager.go:414] "containerMap: removing container" podUID="396d583b-6593-4eff-9eee-7d74a9f4934f" containerName="revoke-keys" Nov 03 19:44:26.002075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002010 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396d583b-6593-4eff-9eee-7d74a9f4934f" containerName="revoke-keys" Nov 03 19:44:26.002075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002054 5233 cpu_manager.go:414] "containerMap: removing container" podUID="1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" containerName="payload-pre-processing" Nov 03 19:44:26.002075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002062 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" containerName="payload-pre-processing" Nov 03 19:44:26.002187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002075 5233 cpu_manager.go:414] "containerMap: removing container" podUID="64e3fd4e-4626-44ab-9406-c2e7c9c92480" containerName="payload-processing" Nov 03 19:44:26.002187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002083 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e3fd4e-4626-44ab-9406-c2e7c9c92480" containerName="payload-processing" Nov 03 19:44:26.002187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002128 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a8236266-058e-429d-a8b4-89e376265d5a" containerName="payload-processing" Nov 03 19:44:26.002187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002135 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8236266-058e-429d-a8b4-89e376265d5a" containerName="payload-processing" Nov 03 19:44:26.002187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002145 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cf8b0cec-42ec-4bb2-83b2-66092f821b2a" containerName="payload-pre-processing" Nov 03 19:44:26.002187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002152 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8b0cec-42ec-4bb2-83b2-66092f821b2a" containerName="payload-pre-processing" Nov 03 19:44:26.002361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002348 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8236266-058e-429d-a8b4-89e376265d5a" containerName="payload-processing" Nov 03 19:44:26.002393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002364 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8b0cec-42ec-4bb2-83b2-66092f821b2a" containerName="payload-pre-processing" Nov 03 19:44:26.002393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002380 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="396d583b-6593-4eff-9eee-7d74a9f4934f" containerName="revoke-keys" Nov 03 19:44:26.002458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002394 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9c03c1-f5c6-4b0c-b59b-96927570e2a3" containerName="payload-pre-processing" Nov 03 19:44:26.002458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.002401 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e3fd4e-4626-44ab-9406-c2e7c9c92480" containerName="payload-processing" Nov 03 19:44:26.028454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.028415 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8"] Nov 03 19:44:26.028696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.028559 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.040649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.040290 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-14a79569\"" Nov 03 19:44:26.040649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.040549 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-14a79569-dockercfg-qtvh6\"" Nov 03 19:44:26.044358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.044327 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.044694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.044675 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hps6f\" (UniqueName: \"kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f\") pod \"payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.061702 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.061672 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj"] Nov 03 19:44:26.095556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.094246 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.190483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.157831 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume\") pod \"payload-processing-e2e-ait-14a79569-7856768d9f-854tj\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.190483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.157927 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hps6f\" (UniqueName: \"kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f\") pod \"payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.190483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.158013 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb68q\" (UniqueName: \"kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q\") pod \"payload-processing-e2e-ait-14a79569-7856768d9f-854tj\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.190483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.158073 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.190483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.167203 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.273013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.268592 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb68q\" (UniqueName: \"kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q\") pod \"payload-processing-e2e-ait-14a79569-7856768d9f-854tj\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.273013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.268992 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume\") pod \"payload-processing-e2e-ait-14a79569-7856768d9f-854tj\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.273013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.270122 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume\") pod \"payload-processing-e2e-ait-14a79569-7856768d9f-854tj\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.299002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.298916 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hps6f\" (UniqueName: \"kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f\") pod \"payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.310169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.310083 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj"] Nov 03 19:44:26.349135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.349013 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb68q\" (UniqueName: \"kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q\") pod \"payload-processing-e2e-ait-14a79569-7856768d9f-854tj\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.377575 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.372221 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:26.662771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.654093 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:26.980068 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:44:26.980038 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb3051b_683d_4c2c_8e64_4e8a7411f60a.slice/crio-46145dbbaeab00b3fc4094aeb355c8f8a4ca0c00e2dd3e6b345f5276a1eeb464 WatchSource:0}: Error finding container 46145dbbaeab00b3fc4094aeb355c8f8a4ca0c00e2dd3e6b345f5276a1eeb464: Status 404 returned error can't find the container with id 46145dbbaeab00b3fc4094aeb355c8f8a4ca0c00e2dd3e6b345f5276a1eeb464 Nov 03 19:44:26.984472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:26.984449 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8"] Nov 03 19:44:27.071932 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:44:27.071859 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065fb1d8_cbf5_4c6d_b29f_62bab5fc1981.slice/crio-50ac7fcf407ddf7cf2a0fb089ea8efa1cc0dfd11f93e3b31ec3b08c77d7676f2 WatchSource:0}: Error finding container 50ac7fcf407ddf7cf2a0fb089ea8efa1cc0dfd11f93e3b31ec3b08c77d7676f2: Status 404 returned error can't find the container with id 50ac7fcf407ddf7cf2a0fb089ea8efa1cc0dfd11f93e3b31ec3b08c77d7676f2 Nov 03 19:44:27.072926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.072899 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj"] Nov 03 19:44:27.960350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.960320 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" event={"ID":"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981","Type":"ContainerStarted","Data":"75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774"} Nov 03 19:44:27.961311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.960369 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" event={"ID":"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981","Type":"ContainerStarted","Data":"50ac7fcf407ddf7cf2a0fb089ea8efa1cc0dfd11f93e3b31ec3b08c77d7676f2"} Nov 03 19:44:27.961311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.960428 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:27.962400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.962374 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" event={"ID":"0eb3051b-683d-4c2c-8e64-4e8a7411f60a","Type":"ContainerStarted","Data":"21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a"} Nov 03 19:44:27.962508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.962423 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" event={"ID":"0eb3051b-683d-4c2c-8e64-4e8a7411f60a","Type":"ContainerStarted","Data":"46145dbbaeab00b3fc4094aeb355c8f8a4ca0c00e2dd3e6b345f5276a1eeb464"} Nov 03 19:44:27.962508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.962452 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:27.981373 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:27.981329 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" podStartSLOduration=1.981313664 podStartE2EDuration="1.981313664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:27.97843775 +0000 UTC m=+1034.319551235" watchObservedRunningTime="2026-11-03 19:44:27.981313664 +0000 UTC m=+1034.322427154" Nov 03 19:44:28.017532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.017477 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" podStartSLOduration=3.017459632 podStartE2EDuration="3.017459632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:28.010356554 +0000 UTC m=+1034.351470041" watchObservedRunningTime="2026-11-03 19:44:28.017459632 +0000 UTC m=+1034.358573117" Nov 03 19:44:28.265662 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.265632 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:44:28.361734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.361701 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.362167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.362146 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.363491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.363459 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.363628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.363581 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.363691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.363624 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.363691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.363669 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.363807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.363776 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.363872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.363823 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.366121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.364043 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token\") pod \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\" (UID: \"fc61436c-04d0-4dbf-aa81-4ea479a5160e\") " Nov 03 19:44:28.366121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.365192 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:44:28.368215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.367716 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:28.368215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.367946 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:44:28.368215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.368136 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:44:28.368215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.368195 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:44:28.412248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.412182 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:28.497708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.470716 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.501053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.501024 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.501279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.501264 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.501397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.501386 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.501509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.501498 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.503661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.503556 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.503836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.472524 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:44:28.503961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.473369 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "kube-api-access-pfvpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:28.504097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.475470 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo" pod "fc61436c-04d0-4dbf-aa81-4ea479a5160e" (UID: "fc61436c-04d0-4dbf-aa81-4ea479a5160e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:44:28.608910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.608871 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.610862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.610828 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc61436c-04d0-4dbf-aa81-4ea479a5160e-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.611121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.611104 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfvpr\" (UniqueName: \"kubernetes.io/projected/fc61436c-04d0-4dbf-aa81-4ea479a5160e-kube-api-access-pfvpr\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:28.976555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.976510 5233 generic.go:356] "Generic (PLEG): container finished" podID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" containerID="fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a" exitCode=0 Nov 03 19:44:28.976751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.976582 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" Nov 03 19:44:28.976751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.976592 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" event={"ID":"fc61436c-04d0-4dbf-aa81-4ea479a5160e","Type":"ContainerDied","Data":"fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a"} Nov 03 19:44:28.976751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.976645 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst" event={"ID":"fc61436c-04d0-4dbf-aa81-4ea479a5160e","Type":"ContainerDied","Data":"fc3f18bc12ebd00cdb7dc730bddb9f0028e289290ec5c9805d055c887a080043"} Nov 03 19:44:28.976751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.976668 5233 scope.go:122] "RemoveContainer" containerID="fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a" Nov 03 19:44:28.989737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.989721 5233 scope.go:122] "RemoveContainer" containerID="fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a" Nov 03 19:44:28.990072 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:28.990013 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a\": container with ID starting with fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a not found: ID does not exist" containerID="fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a" Nov 03 19:44:28.990072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:28.990037 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a"} err="failed to get container status \"fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a\": rpc error: code = NotFound desc = could not find container \"fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a\": container with ID starting with fe3c0da38c35b8e3f7dd8b4bbfe365ead378f3bba813ea0bf674253166aea88a not found: ID does not exist" Nov 03 19:44:29.023367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:29.023342 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst"] Nov 03 19:44:29.041986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:29.041947 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ca12e9fb-gw-openshift-default-66474c97bf-w8tst"] Nov 03 19:44:29.861018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:29.860955 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" path="/var/lib/kubelet/pods/fc61436c-04d0-4dbf-aa81-4ea479a5160e/volumes" Nov 03 19:44:31.253580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.253534 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j"] Nov 03 19:44:31.260207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.260162 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc935cf-xhc2j"] Nov 03 19:44:31.878099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.877346 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396d583b-6593-4eff-9eee-7d74a9f4934f" path="/var/lib/kubelet/pods/396d583b-6593-4eff-9eee-7d74a9f4934f/volumes" Nov 03 19:44:31.994035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.992182 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv"] Nov 03 19:44:31.994035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.993041 5233 cpu_manager.go:414] "containerMap: removing container" podUID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" containerName="istio-proxy" Nov 03 19:44:31.994035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.993057 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" containerName="istio-proxy" Nov 03 19:44:31.994035 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:31.993231 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc61436c-04d0-4dbf-aa81-4ea479a5160e" containerName="istio-proxy" Nov 03 19:44:32.008990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.006432 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.017256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.011649 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:44:32.017256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.011795 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:44:32.035419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.035358 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv"] Nov 03 19:44:32.117820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.117789 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcp8\" (UniqueName: \"kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.118037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.117936 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.118131 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.118116 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.219723 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.219685 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.219900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.219785 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcp8\" (UniqueName: \"kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.219966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.219949 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.220538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.220516 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.222312 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.222292 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.233024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.233001 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcp8\" (UniqueName: \"kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8\") pod \"maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.346387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.346352 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:32.674680 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:44:32.674648 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6342e14_de0d_4b52_8d00_14efb0225bdb.slice/crio-105ff1136fa551b97a7daaaa69ac00379ea63935e6db3482c06b6b20aad04dcc WatchSource:0}: Error finding container 105ff1136fa551b97a7daaaa69ac00379ea63935e6db3482c06b6b20aad04dcc: Status 404 returned error can't find the container with id 105ff1136fa551b97a7daaaa69ac00379ea63935e6db3482c06b6b20aad04dcc Nov 03 19:44:32.676445 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:32.676421 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv"] Nov 03 19:44:33.067298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:33.067262 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" event={"ID":"e6342e14-de0d-4b52-8d00-14efb0225bdb","Type":"ContainerStarted","Data":"105ff1136fa551b97a7daaaa69ac00379ea63935e6db3482c06b6b20aad04dcc"} Nov 03 19:44:34.097242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:34.097149 5233 generic.go:356] "Generic (PLEG): container finished" podID="e6342e14-de0d-4b52-8d00-14efb0225bdb" containerID="ab6b2637f3f49997a9a796d27704282534d2cab4f923d8fd67aefdfc95a71f55" exitCode=0 Nov 03 19:44:34.097242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:34.097186 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" event={"ID":"e6342e14-de0d-4b52-8d00-14efb0225bdb","Type":"ContainerDied","Data":"ab6b2637f3f49997a9a796d27704282534d2cab4f923d8fd67aefdfc95a71f55"} Nov 03 19:44:35.302277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.302253 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:35.382935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.382845 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8\" (UniqueName: \"kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8\") pod \"e6342e14-de0d-4b52-8d00-14efb0225bdb\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " Nov 03 19:44:35.383137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.382994 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca\") pod \"e6342e14-de0d-4b52-8d00-14efb0225bdb\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " Nov 03 19:44:35.383137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.383050 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca\") pod \"e6342e14-de0d-4b52-8d00-14efb0225bdb\" (UID: \"e6342e14-de0d-4b52-8d00-14efb0225bdb\") " Nov 03 19:44:35.383778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.383754 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca" pod "e6342e14-de0d-4b52-8d00-14efb0225bdb" (UID: "e6342e14-de0d-4b52-8d00-14efb0225bdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:35.386602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.386574 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8" pod "e6342e14-de0d-4b52-8d00-14efb0225bdb" (UID: "e6342e14-de0d-4b52-8d00-14efb0225bdb"). InnerVolumeSpecName "kube-api-access-tmcp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:35.386836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.386800 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca" pod "e6342e14-de0d-4b52-8d00-14efb0225bdb" (UID: "e6342e14-de0d-4b52-8d00-14efb0225bdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:35.485531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.485499 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:35.485531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.485528 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6342e14-de0d-4b52-8d00-14efb0225bdb-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:35.485531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:35.485539 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcp8\" (UniqueName: \"kubernetes.io/projected/e6342e14-de0d-4b52-8d00-14efb0225bdb-kube-api-access-tmcp8\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:36.062898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.062862 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb"] Nov 03 19:44:36.063786 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.063765 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e6342e14-de0d-4b52-8d00-14efb0225bdb" containerName="revoke-keys" Nov 03 19:44:36.063837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.063788 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6342e14-de0d-4b52-8d00-14efb0225bdb" containerName="revoke-keys" Nov 03 19:44:36.063916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.063907 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6342e14-de0d-4b52-8d00-14efb0225bdb" containerName="revoke-keys" Nov 03 19:44:36.069421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.069393 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.074202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.074175 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb"] Nov 03 19:44:36.168843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.168699 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" event={"ID":"e6342e14-de0d-4b52-8d00-14efb0225bdb","Type":"ContainerDied","Data":"105ff1136fa551b97a7daaaa69ac00379ea63935e6db3482c06b6b20aad04dcc"} Nov 03 19:44:36.169491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.169466 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv" Nov 03 19:44:36.170600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.170579 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105ff1136fa551b97a7daaaa69ac00379ea63935e6db3482c06b6b20aad04dcc" Nov 03 19:44:36.228995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.223767 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.228995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.223866 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txn9g\" (UniqueName: \"kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.228995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.224069 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.333994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.333188 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txn9g\" (UniqueName: \"kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.335316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.335288 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.338710 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.335741 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.342381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.339735 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.343094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.342944 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.348545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.348522 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txn9g\" (UniqueName: \"kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g\") pod \"maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.438565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.438486 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:36.774942 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:44:36.774899 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764338d2_8bcf_4113_b1a2_db9cde207872.slice/crio-630b1b69fea980f5418332e39e01a622569422d9769d8d472056ba510e3fe36a WatchSource:0}: Error finding container 630b1b69fea980f5418332e39e01a622569422d9769d8d472056ba510e3fe36a: Status 404 returned error can't find the container with id 630b1b69fea980f5418332e39e01a622569422d9769d8d472056ba510e3fe36a Nov 03 19:44:36.806831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:36.806712 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb"] Nov 03 19:44:37.188698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:37.188654 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" event={"ID":"764338d2-8bcf-4113-b1a2-db9cde207872","Type":"ContainerStarted","Data":"630b1b69fea980f5418332e39e01a622569422d9769d8d472056ba510e3fe36a"} Nov 03 19:44:38.204922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:38.204824 5233 generic.go:356] "Generic (PLEG): container finished" podID="764338d2-8bcf-4113-b1a2-db9cde207872" containerID="10e57cb0e965455a4aecb00185615d671c8233bb04ce2e125f67e3f75ffecf4d" exitCode=0 Nov 03 19:44:38.205348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:38.204912 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" event={"ID":"764338d2-8bcf-4113-b1a2-db9cde207872","Type":"ContainerDied","Data":"10e57cb0e965455a4aecb00185615d671c8233bb04ce2e125f67e3f75ffecf4d"} Nov 03 19:44:38.980818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:38.980792 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:44:38.981050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:38.980856 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:44:39.427866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.427836 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:39.531983 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.531879 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g\" (UniqueName: \"kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g\") pod \"764338d2-8bcf-4113-b1a2-db9cde207872\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " Nov 03 19:44:39.532329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.532307 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca\") pod \"764338d2-8bcf-4113-b1a2-db9cde207872\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " Nov 03 19:44:39.532411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.532362 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca\") pod \"764338d2-8bcf-4113-b1a2-db9cde207872\" (UID: \"764338d2-8bcf-4113-b1a2-db9cde207872\") " Nov 03 19:44:39.532802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.532747 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca" pod "764338d2-8bcf-4113-b1a2-db9cde207872" (UID: "764338d2-8bcf-4113-b1a2-db9cde207872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:39.533137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.533111 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:39.534945 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.534917 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca" pod "764338d2-8bcf-4113-b1a2-db9cde207872" (UID: "764338d2-8bcf-4113-b1a2-db9cde207872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:39.539814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.539793 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g" pod "764338d2-8bcf-4113-b1a2-db9cde207872" (UID: "764338d2-8bcf-4113-b1a2-db9cde207872"). InnerVolumeSpecName "kube-api-access-txn9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:39.644003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.643784 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764338d2-8bcf-4113-b1a2-db9cde207872-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:39.644003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:39.643830 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txn9g\" (UniqueName: \"kubernetes.io/projected/764338d2-8bcf-4113-b1a2-db9cde207872-kube-api-access-txn9g\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:40.248131 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:40.248103 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" Nov 03 19:44:40.248321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:40.248146 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb" event={"ID":"764338d2-8bcf-4113-b1a2-db9cde207872","Type":"ContainerDied","Data":"630b1b69fea980f5418332e39e01a622569422d9769d8d472056ba510e3fe36a"} Nov 03 19:44:40.248321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:40.248209 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="630b1b69fea980f5418332e39e01a622569422d9769d8d472056ba510e3fe36a" Nov 03 19:44:42.128935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:42.128887 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn"] Nov 03 19:44:46.196579 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:46.196528 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb"] Nov 03 19:44:47.132992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.132026 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" podUID="8d1b9326-e198-47dd-bb52-f067f4173d74" containerName="payload-processing" containerID="cri-o://006874786b4105f60af4e1a8ebc84ce7f648642d4d7d3256efe61780510eaec1" gracePeriod=25 Nov 03 19:44:47.383229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.383144 5233 generic.go:356] "Generic (PLEG): container finished" podID="8d1b9326-e198-47dd-bb52-f067f4173d74" containerID="006874786b4105f60af4e1a8ebc84ce7f648642d4d7d3256efe61780510eaec1" exitCode=0 Nov 03 19:44:47.383762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.383383 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" event={"ID":"8d1b9326-e198-47dd-bb52-f067f4173d74","Type":"ContainerDied","Data":"006874786b4105f60af4e1a8ebc84ce7f648642d4d7d3256efe61780510eaec1"} Nov 03 19:44:47.513891 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.513863 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:44:47.680882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.680790 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume\") pod \"8d1b9326-e198-47dd-bb52-f067f4173d74\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " Nov 03 19:44:47.681070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.681042 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6\" (UniqueName: \"kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6\") pod \"8d1b9326-e198-47dd-bb52-f067f4173d74\" (UID: \"8d1b9326-e198-47dd-bb52-f067f4173d74\") " Nov 03 19:44:47.681206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.681180 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume" pod "8d1b9326-e198-47dd-bb52-f067f4173d74" (UID: "8d1b9326-e198-47dd-bb52-f067f4173d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:47.681786 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.681761 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1b9326-e198-47dd-bb52-f067f4173d74-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:47.683454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.683432 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6" pod "8d1b9326-e198-47dd-bb52-f067f4173d74" (UID: "8d1b9326-e198-47dd-bb52-f067f4173d74"). InnerVolumeSpecName "kube-api-access-tx7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:47.787735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:47.787687 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7b6\" (UniqueName: \"kubernetes.io/projected/8d1b9326-e198-47dd-bb52-f067f4173d74-kube-api-access-tx7b6\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:48.471047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.470632 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" event={"ID":"8d1b9326-e198-47dd-bb52-f067f4173d74","Type":"ContainerDied","Data":"628a66248c5bf2a5537c5b668bdda16c58e0c07b589f63ccf84669e8c166825c"} Nov 03 19:44:48.471047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.470748 5233 scope.go:122] "RemoveContainer" containerID="006874786b4105f60af4e1a8ebc84ce7f648642d4d7d3256efe61780510eaec1" Nov 03 19:44:48.471047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.470987 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn" Nov 03 19:44:48.625749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.625717 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn"] Nov 03 19:44:48.647205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.647177 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv"] Nov 03 19:44:48.658853 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.658791 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28ef3ba9-845b5dbd4c-jvwvn"] Nov 03 19:44:48.670764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:48.670713 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28ef3ba9-jgbbv"] Nov 03 19:44:49.853087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:49.853054 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1b9326-e198-47dd-bb52-f067f4173d74" path="/var/lib/kubelet/pods/8d1b9326-e198-47dd-bb52-f067f4173d74/volumes" Nov 03 19:44:49.854412 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:49.854388 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6342e14-de0d-4b52-8d00-14efb0225bdb" path="/var/lib/kubelet/pods/e6342e14-de0d-4b52-8d00-14efb0225bdb/volumes" Nov 03 19:44:51.108079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.108038 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7"] Nov 03 19:44:51.109861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.109836 5233 cpu_manager.go:414] "containerMap: removing container" podUID="764338d2-8bcf-4113-b1a2-db9cde207872" containerName="revoke-keys" Nov 03 19:44:51.110056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.110042 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764338d2-8bcf-4113-b1a2-db9cde207872" containerName="revoke-keys" Nov 03 19:44:51.110418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.110401 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8d1b9326-e198-47dd-bb52-f067f4173d74" containerName="payload-processing" Nov 03 19:44:51.110580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.110567 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1b9326-e198-47dd-bb52-f067f4173d74" containerName="payload-processing" Nov 03 19:44:51.110924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.110907 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="764338d2-8bcf-4113-b1a2-db9cde207872" containerName="revoke-keys" Nov 03 19:44:51.111054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.111043 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1b9326-e198-47dd-bb52-f067f4173d74" containerName="payload-processing" Nov 03 19:44:51.194764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.117529 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.194764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.159571 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:44:51.194764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.159687 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:44:51.278007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.255340 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" podUID="9db8eeca-6fed-490e-9f00-c859998d4876" containerName="payload-processing" containerID="cri-o://f13ed035c6c84020fe6eb774a74e8e348ca0b60bc08b54f0eaf54bfa0d0dc76b" gracePeriod=25 Nov 03 19:44:51.278007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.255677 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7"] Nov 03 19:44:51.297015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.293890 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4lw\" (UniqueName: \"kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.297015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.293956 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.297015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.294139 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.485350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.415070 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4lw\" (UniqueName: \"kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.485350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.415172 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.485350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.441493 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.500305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:51.487406 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:51.610074 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:44:51.607407 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:44:52.095996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.095717 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:44:52.324000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.308926 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:52.466073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.319760 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4lw\" (UniqueName: \"kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw\") pod \"maas-api-revoke-keys-e2e-ait-14a79569-h97n7\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:52.470349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.375736 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:52.515645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.438699 5233 generic.go:356] "Generic (PLEG): container finished" podID="9db8eeca-6fed-490e-9f00-c859998d4876" containerID="f13ed035c6c84020fe6eb774a74e8e348ca0b60bc08b54f0eaf54bfa0d0dc76b" exitCode=0 Nov 03 19:44:52.516295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.445883 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" event={"ID":"9db8eeca-6fed-490e-9f00-c859998d4876","Type":"ContainerDied","Data":"f13ed035c6c84020fe6eb774a74e8e348ca0b60bc08b54f0eaf54bfa0d0dc76b"} Nov 03 19:44:52.556005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.555829 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:44:52.592256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.592227 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:52.722624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.721029 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb"] Nov 03 19:44:52.730021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.723905 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:44:52.734648 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.732998 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b7a388c-cvnwb"] Nov 03 19:44:52.826996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.775831 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:52.898049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.892435 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:52.947025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:52.940481 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:53.129703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.129642 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:53.130424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.130399 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:44:53.586013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.585055 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:44:53.586013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.585345 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:44:53.681624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.670611 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:44:53.722831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.696735 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:53.793076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.780099 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:44:53.793076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.781771 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:44:53.871630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:53.871585 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:54.008752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.008710 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.260634 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261038 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764338d2-8bcf-4113-b1a2-db9cde207872" path="/var/lib/kubelet/pods/764338d2-8bcf-4113-b1a2-db9cde207872/volumes" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261545 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261689 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261720 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261746 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261773 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:44:54.262346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.261800 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:44:54.320995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.311112 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7"] Nov 03 19:44:54.446740 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.446711 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume\") pod \"9db8eeca-6fed-490e-9f00-c859998d4876\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " Nov 03 19:44:54.446939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.446819 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649\" (UniqueName: \"kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649\") pod \"9db8eeca-6fed-490e-9f00-c859998d4876\" (UID: \"9db8eeca-6fed-490e-9f00-c859998d4876\") " Nov 03 19:44:54.448900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.447463 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume" pod "9db8eeca-6fed-490e-9f00-c859998d4876" (UID: "9db8eeca-6fed-490e-9f00-c859998d4876"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:54.448900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.447714 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db8eeca-6fed-490e-9f00-c859998d4876-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:54.449949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.449922 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649" pod "9db8eeca-6fed-490e-9f00-c859998d4876" (UID: "9db8eeca-6fed-490e-9f00-c859998d4876"). InnerVolumeSpecName "kube-api-access-85649". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:54.548874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.548783 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85649\" (UniqueName: \"kubernetes.io/projected/9db8eeca-6fed-490e-9f00-c859998d4876-kube-api-access-85649\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:54.936321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.936218 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" event={"ID":"6974d641-034c-425a-bfea-d6a88f35a552","Type":"ContainerStarted","Data":"bfc6034ba51c5611b82d05716468cc641a235324016d00c461e673cce4bd1358"} Nov 03 19:44:54.940277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.940244 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" event={"ID":"6974d641-034c-425a-bfea-d6a88f35a552","Type":"ContainerStarted","Data":"c7fbd2d0a4381f15dbbbd1a9a2c531e255a90e8aa2c6e4ce8f21d7f85989c48c"} Nov 03 19:44:54.948613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.942490 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" Nov 03 19:44:54.948613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.942572 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb" event={"ID":"9db8eeca-6fed-490e-9f00-c859998d4876","Type":"ContainerDied","Data":"7b8e6eec3daa93d3a4a323221d9bdf19fef4f63f653ce7b12edc845c844f0dde"} Nov 03 19:44:54.948613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.942646 5233 scope.go:122] "RemoveContainer" containerID="f13ed035c6c84020fe6eb774a74e8e348ca0b60bc08b54f0eaf54bfa0d0dc76b" Nov 03 19:44:54.977389 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:54.977347 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" podStartSLOduration=3.977326358 podStartE2EDuration="3.977326358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:54.972851366 +0000 UTC m=+1061.313964854" watchObservedRunningTime="2026-11-03 19:44:54.977326358 +0000 UTC m=+1061.318439843" Nov 03 19:44:55.019952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:55.019914 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb"] Nov 03 19:44:55.030852 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:55.030816 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b7a388c-54bfcd894f-gl8rb"] Nov 03 19:44:55.847060 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:55.846918 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db8eeca-6fed-490e-9f00-c859998d4876" path="/var/lib/kubelet/pods/9db8eeca-6fed-490e-9f00-c859998d4876/volumes" Nov 03 19:44:55.957142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:55.957110 5233 generic.go:356] "Generic (PLEG): container finished" podID="6974d641-034c-425a-bfea-d6a88f35a552" containerID="bfc6034ba51c5611b82d05716468cc641a235324016d00c461e673cce4bd1358" exitCode=0 Nov 03 19:44:55.957637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:55.957191 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" event={"ID":"6974d641-034c-425a-bfea-d6a88f35a552","Type":"ContainerDied","Data":"bfc6034ba51c5611b82d05716468cc641a235324016d00c461e673cce4bd1358"} Nov 03 19:44:57.162550 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.162525 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:57.184322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.184295 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca\") pod \"6974d641-034c-425a-bfea-d6a88f35a552\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " Nov 03 19:44:57.184475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.184373 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw\" (UniqueName: \"kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw\") pod \"6974d641-034c-425a-bfea-d6a88f35a552\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " Nov 03 19:44:57.184533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.184507 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca\") pod \"6974d641-034c-425a-bfea-d6a88f35a552\" (UID: \"6974d641-034c-425a-bfea-d6a88f35a552\") " Nov 03 19:44:57.184747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.184716 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca" pod "6974d641-034c-425a-bfea-d6a88f35a552" (UID: "6974d641-034c-425a-bfea-d6a88f35a552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:44:57.185428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.185403 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:57.188303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.188109 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca" pod "6974d641-034c-425a-bfea-d6a88f35a552" (UID: "6974d641-034c-425a-bfea-d6a88f35a552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:57.188485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.188454 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw" pod "6974d641-034c-425a-bfea-d6a88f35a552" (UID: "6974d641-034c-425a-bfea-d6a88f35a552"). InnerVolumeSpecName "kube-api-access-ff4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:57.287881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.287801 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff4lw\" (UniqueName: \"kubernetes.io/projected/6974d641-034c-425a-bfea-d6a88f35a552-kube-api-access-ff4lw\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:57.287881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.287841 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974d641-034c-425a-bfea-d6a88f35a552-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:57.985784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.985753 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" Nov 03 19:44:57.985988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.985794 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7" event={"ID":"6974d641-034c-425a-bfea-d6a88f35a552","Type":"ContainerDied","Data":"c7fbd2d0a4381f15dbbbd1a9a2c531e255a90e8aa2c6e4ce8f21d7f85989c48c"} Nov 03 19:44:57.985988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:57.985852 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7fbd2d0a4381f15dbbbd1a9a2c531e255a90e8aa2c6e4ce8f21d7f85989c48c" Nov 03 19:44:58.195956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.195869 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v"] Nov 03 19:44:58.230001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.220316 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9db8eeca-6fed-490e-9f00-c859998d4876" containerName="payload-processing" Nov 03 19:44:58.230001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.220347 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db8eeca-6fed-490e-9f00-c859998d4876" containerName="payload-processing" Nov 03 19:44:58.230001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.220462 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6974d641-034c-425a-bfea-d6a88f35a552" containerName="revoke-keys" Nov 03 19:44:58.230001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.220470 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6974d641-034c-425a-bfea-d6a88f35a552" containerName="revoke-keys" Nov 03 19:44:58.230001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.220835 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db8eeca-6fed-490e-9f00-c859998d4876" containerName="payload-processing" Nov 03 19:44:58.230001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.220869 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6974d641-034c-425a-bfea-d6a88f35a552" containerName="revoke-keys" Nov 03 19:44:58.235986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.234120 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.239290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.239266 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c54f85-openshift-default-dockercfg-w8jpj\"" Nov 03 19:44:58.241242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.241212 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v"] Nov 03 19:44:58.350023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.347166 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373269 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373355 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373389 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373430 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373512 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5gq\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373625 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373712 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.374995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.373839 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480070 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480362 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480201 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480362 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480260 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480364 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480419 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480447 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480486 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480557 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5gq\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.480713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.480677 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.482057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.482025 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.482309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.482286 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.483340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.483313 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.508049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.496744 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.508049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.497408 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.514264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.514235 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.514437 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.514419 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.519134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.519083 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.523345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.523304 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5gq\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.552939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.552615 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:58.884332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.884295 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:58.884470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.884391 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:58.884470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.884436 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:44:58.884764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.884731 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v"] Nov 03 19:44:58.999192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.999160 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" event={"ID":"153c41c1-4143-4168-883d-a6b1af731ce6","Type":"ContainerStarted","Data":"a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd"} Nov 03 19:44:58.999377 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:58.999228 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" event={"ID":"153c41c1-4143-4168-883d-a6b1af731ce6","Type":"ContainerStarted","Data":"124339e5b82b7cc595c52830839b7fefbef9d8ed1046dff5e8661ef5b9c54880"} Nov 03 19:44:59.032316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:59.032266 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podStartSLOduration=1.032252139 podStartE2EDuration="1.032252139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:59.023339485 +0000 UTC m=+1065.364452971" watchObservedRunningTime="2026-11-03 19:44:59.032252139 +0000 UTC m=+1065.373365624" Nov 03 19:44:59.554068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:59.554040 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:44:59.560850 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:44:59.560824 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:45:00.018268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.018235 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:45:00.020737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.020710 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:45:00.062067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.062025 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l"] Nov 03 19:45:00.078826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.078785 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr"] Nov 03 19:45:00.092603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.092577 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt"] Nov 03 19:45:00.106992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.101349 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1b5af6-j5w7l"] Nov 03 19:45:00.125260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.125228 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c31c60-l5tbr"] Nov 03 19:45:00.132781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.132754 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df422c42-c8vtt"] Nov 03 19:45:00.157332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.157307 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx"] Nov 03 19:45:00.172572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.168938 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.184933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.184825 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx"] Nov 03 19:45:00.191532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.191455 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:45:00.243278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.243234 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh"] Nov 03 19:45:00.245486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.244644 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.266835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.266800 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn"] Nov 03 19:45:00.267081 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.266901 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.295091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.295065 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx"] Nov 03 19:45:00.295256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.295117 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx"] Nov 03 19:45:00.295256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.295144 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn"] Nov 03 19:45:00.295256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.295169 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.299721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.299697 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c"] Nov 03 19:45:00.312192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.308174 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.312192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.310138 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x"] Nov 03 19:45:00.315950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.315928 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.318333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.318286 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c"] Nov 03 19:45:00.325537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.325263 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh"] Nov 03 19:45:00.341564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.341542 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x"] Nov 03 19:45:00.341957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.341937 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbvf\" (UniqueName: \"kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf\") pod \"maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.342281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.342261 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.342418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.342350 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.342505 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.342488 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwx8k\" (UniqueName: \"kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k\") pod \"maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.444307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444276 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x854\" (UniqueName: \"kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854\") pod \"maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.444483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444413 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwx8k\" (UniqueName: \"kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k\") pod \"maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.444483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444442 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktpx\" (UniqueName: \"kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx\") pod \"maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.444483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444467 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-zkssn\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.444609 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444514 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.444731 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444709 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbvf\" (UniqueName: \"kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf\") pod \"maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.444789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444776 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlmt\" (UniqueName: \"kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt\") pod \"maas-api-key-cleanup-29895585-zkssn\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.444844 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444824 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.444913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444897 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.444987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.444927 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6h5g\" (UniqueName: \"kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g\") pod \"maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.445044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.445026 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.445112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.445054 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.445584 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.445567 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.445836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.445809 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.460172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.460146 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbvf\" (UniqueName: \"kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf\") pod \"maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.460729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.460706 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwx8k\" (UniqueName: \"kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k\") pod \"maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550242 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktpx\" (UniqueName: \"kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx\") pod \"maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550297 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-zkssn\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550360 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550494 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlmt\" (UniqueName: \"kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt\") pod \"maas-api-key-cleanup-29895585-zkssn\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550553 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550592 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6h5g\" (UniqueName: \"kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g\") pod \"maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550676 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.550824 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x854\" (UniqueName: \"kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854\") pod \"maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.552041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.551478 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:00.554454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.553518 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.554454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.554171 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.554926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.554762 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.559302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.558667 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-zkssn\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.573995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.568533 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktpx\" (UniqueName: \"kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx\") pod \"maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.573995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.569093 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x854\" (UniqueName: \"kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854\") pod \"maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.573995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.571644 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlmt\" (UniqueName: \"kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt\") pod \"maas-api-key-cleanup-29895585-zkssn\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.582686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.580678 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6h5g\" (UniqueName: \"kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g\") pod \"maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:00.590095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.590024 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:00.600675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.600538 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:00.613330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.613302 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:00.633362 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.630064 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:00.638781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:00.638560 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:01.043191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.043158 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn"] Nov 03 19:45:01.054443 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.054413 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b723e127-5x4jn"] Nov 03 19:45:01.137664 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.131055 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx"] Nov 03 19:45:01.270318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.270239 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj"] Nov 03 19:45:01.295539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.295379 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8"] Nov 03 19:45:01.363660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.363627 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx"] Nov 03 19:45:01.414833 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:01.412498 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8b1ba6_0b40_4ed2_a47e_dfa99a32aa76.slice/crio-942fed3709b710be2cadda3a833cbde171297679da610cdf73041211edb873b8 WatchSource:0}: Error finding container 942fed3709b710be2cadda3a833cbde171297679da610cdf73041211edb873b8: Status 404 returned error can't find the container with id 942fed3709b710be2cadda3a833cbde171297679da610cdf73041211edb873b8 Nov 03 19:45:01.447435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.447391 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn"] Nov 03 19:45:01.453018 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:01.452667 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cb8835_75ed_4d1c_9c2d_7e21a8e671b8.slice/crio-a27b0ccb45d1c0844025cc0ee3ef06f997dba57838c3f048651b6b581ed8c128 WatchSource:0}: Error finding container a27b0ccb45d1c0844025cc0ee3ef06f997dba57838c3f048651b6b581ed8c128: Status 404 returned error can't find the container with id a27b0ccb45d1c0844025cc0ee3ef06f997dba57838c3f048651b6b581ed8c128 Nov 03 19:45:01.462256 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.462171 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh"] Nov 03 19:45:01.479334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.475141 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c"] Nov 03 19:45:01.520504 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:01.520442 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cc15f2_4f17_4d23_a3f4_dae31e05a6b8.slice/crio-7c3783b10c64c6107fd9a3781e8b85558ee05026a17113b9a67dc4950f19ec5f WatchSource:0}: Error finding container 7c3783b10c64c6107fd9a3781e8b85558ee05026a17113b9a67dc4950f19ec5f: Status 404 returned error can't find the container with id 7c3783b10c64c6107fd9a3781e8b85558ee05026a17113b9a67dc4950f19ec5f Nov 03 19:45:01.523507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.523483 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x"] Nov 03 19:45:01.869699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.869662 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c" path="/var/lib/kubelet/pods/3e19e8dd-3e5e-41b1-9c4f-12ce9b5b033c/volumes" Nov 03 19:45:01.870448 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.870426 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c984a88b-530d-4fb8-8b3f-46a11341bbce" path="/var/lib/kubelet/pods/c984a88b-530d-4fb8-8b3f-46a11341bbce/volumes" Nov 03 19:45:01.871401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.871348 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da782a86-b292-4ed6-af5a-cb40161cb55a" path="/var/lib/kubelet/pods/da782a86-b292-4ed6-af5a-cb40161cb55a/volumes" Nov 03 19:45:01.872650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:01.872627 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f440ca15-e099-4938-a428-34cbda00686d" path="/var/lib/kubelet/pods/f440ca15-e099-4938-a428-34cbda00686d/volumes" Nov 03 19:45:02.065569 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.065454 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerStarted","Data":"ab2d64dab1b173675a5cebb5639e62cb7de7666eeeab311ecfbb0804bb0f6cc4"} Nov 03 19:45:02.065569 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.065518 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerStarted","Data":"7c3783b10c64c6107fd9a3781e8b85558ee05026a17113b9a67dc4950f19ec5f"} Nov 03 19:45:02.068010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.067885 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerStarted","Data":"d0d19b26bee2552da0bc6b3e57702851c1bec083356e03a4420b527d9d5819ef"} Nov 03 19:45:02.068010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.067936 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerStarted","Data":"ba3f2fa89c68973e58b3417299c22cd55d68c94f4210c77457c9d49c6b03262a"} Nov 03 19:45:02.072583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.072188 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" event={"ID":"052ebdb6-5f1a-4923-ad1d-98a1150991d7","Type":"ContainerStarted","Data":"23345ff0669c1c196a03267a8bbae4200dab8f8529e1e0fd7e99eae8cf9dacc9"} Nov 03 19:45:02.072583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.072236 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" event={"ID":"052ebdb6-5f1a-4923-ad1d-98a1150991d7","Type":"ContainerStarted","Data":"529e214345bfec811f8f74dcc0c4345e0f4cf56e72accb4e0e07b0ea9666310f"} Nov 03 19:45:02.073793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.073762 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" event={"ID":"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8","Type":"ContainerStarted","Data":"a27b0ccb45d1c0844025cc0ee3ef06f997dba57838c3f048651b6b581ed8c128"} Nov 03 19:45:02.075711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.075685 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerStarted","Data":"fffe67a2157b04397165cde828dd7da4abc7fa5825cbff4be72364899a47b333"} Nov 03 19:45:02.075823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.075730 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerStarted","Data":"9188cdbffa4cc0e51d87eae9f124c7399ab16e005dc0c362457e002b7b3527bf"} Nov 03 19:45:02.079068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.079040 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerStarted","Data":"c857920a23c039169c0c447f032b018b004562163dddc10690c986515c377107"} Nov 03 19:45:02.079169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.079085 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerStarted","Data":"942fed3709b710be2cadda3a833cbde171297679da610cdf73041211edb873b8"} Nov 03 19:45:02.097725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.096889 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" podStartSLOduration=2.096869407 podStartE2EDuration="2.096869407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:02.087902443 +0000 UTC m=+1068.429015930" watchObservedRunningTime="2026-11-03 19:45:02.096869407 +0000 UTC m=+1068.437982893" Nov 03 19:45:02.114866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.114819 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" podStartSLOduration=2.114798761 podStartE2EDuration="2.114798761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:02.107211091 +0000 UTC m=+1068.448324581" watchObservedRunningTime="2026-11-03 19:45:02.114798761 +0000 UTC m=+1068.455912259" Nov 03 19:45:02.165576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.165524 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" podStartSLOduration=2.16550951 podStartE2EDuration="2.16550951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:02.159431321 +0000 UTC m=+1068.500544806" watchObservedRunningTime="2026-11-03 19:45:02.16550951 +0000 UTC m=+1068.506623026" Nov 03 19:45:02.272352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.272302 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" podStartSLOduration=2.272288272 podStartE2EDuration="2.272288272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:02.268044125 +0000 UTC m=+1068.609157610" watchObservedRunningTime="2026-11-03 19:45:02.272288272 +0000 UTC m=+1068.613401757" Nov 03 19:45:02.272957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.272931 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" podStartSLOduration=2.272923343 podStartE2EDuration="2.272923343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:02.21604239 +0000 UTC m=+1068.557155875" watchObservedRunningTime="2026-11-03 19:45:02.272923343 +0000 UTC m=+1068.614036827" Nov 03 19:45:02.775816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.775784 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4"] Nov 03 19:45:02.780778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.780760 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:02.786094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.786067 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c54f85-dockercfg-p28cz\"" Nov 03 19:45:02.816586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.816555 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4"] Nov 03 19:45:02.831594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:02.831565 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c54f85\"" Nov 03 19:45:03.082032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.075134 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpwl\" (UniqueName: \"kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl\") pod \"payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.082032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.075385 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.123161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.123123 5233 generic.go:356] "Generic (PLEG): container finished" podID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerID="d0d19b26bee2552da0bc6b3e57702851c1bec083356e03a4420b527d9d5819ef" exitCode=60 Nov 03 19:45:03.123357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.123235 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerDied","Data":"d0d19b26bee2552da0bc6b3e57702851c1bec083356e03a4420b527d9d5819ef"} Nov 03 19:45:03.123783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.123761 5233 scope.go:122] "RemoveContainer" containerID="d0d19b26bee2552da0bc6b3e57702851c1bec083356e03a4420b527d9d5819ef" Nov 03 19:45:03.127017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.126922 5233 generic.go:356] "Generic (PLEG): container finished" podID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerID="23345ff0669c1c196a03267a8bbae4200dab8f8529e1e0fd7e99eae8cf9dacc9" exitCode=60 Nov 03 19:45:03.127192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.127042 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" event={"ID":"052ebdb6-5f1a-4923-ad1d-98a1150991d7","Type":"ContainerDied","Data":"23345ff0669c1c196a03267a8bbae4200dab8f8529e1e0fd7e99eae8cf9dacc9"} Nov 03 19:45:03.140911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.139751 5233 generic.go:356] "Generic (PLEG): container finished" podID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerID="608b03ac00d178e86b7038ac3bf4bef6c8f4080e2e2d7919148091bd6342cabb" exitCode=60 Nov 03 19:45:03.140911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.139831 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" event={"ID":"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8","Type":"ContainerDied","Data":"608b03ac00d178e86b7038ac3bf4bef6c8f4080e2e2d7919148091bd6342cabb"} Nov 03 19:45:03.140911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.140329 5233 scope.go:122] "RemoveContainer" containerID="608b03ac00d178e86b7038ac3bf4bef6c8f4080e2e2d7919148091bd6342cabb" Nov 03 19:45:03.141476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.141454 5233 scope.go:122] "RemoveContainer" containerID="23345ff0669c1c196a03267a8bbae4200dab8f8529e1e0fd7e99eae8cf9dacc9" Nov 03 19:45:03.146031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.144781 5233 generic.go:356] "Generic (PLEG): container finished" podID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerID="fffe67a2157b04397165cde828dd7da4abc7fa5825cbff4be72364899a47b333" exitCode=60 Nov 03 19:45:03.146031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.144879 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerDied","Data":"fffe67a2157b04397165cde828dd7da4abc7fa5825cbff4be72364899a47b333"} Nov 03 19:45:03.146315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.146292 5233 scope.go:122] "RemoveContainer" containerID="fffe67a2157b04397165cde828dd7da4abc7fa5825cbff4be72364899a47b333" Nov 03 19:45:03.147904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.147881 5233 generic.go:356] "Generic (PLEG): container finished" podID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerID="c857920a23c039169c0c447f032b018b004562163dddc10690c986515c377107" exitCode=60 Nov 03 19:45:03.148059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.148037 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerDied","Data":"c857920a23c039169c0c447f032b018b004562163dddc10690c986515c377107"} Nov 03 19:45:03.148359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.148315 5233 scope.go:122] "RemoveContainer" containerID="c857920a23c039169c0c447f032b018b004562163dddc10690c986515c377107" Nov 03 19:45:03.151318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.151301 5233 generic.go:356] "Generic (PLEG): container finished" podID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerID="ab2d64dab1b173675a5cebb5639e62cb7de7666eeeab311ecfbb0804bb0f6cc4" exitCode=60 Nov 03 19:45:03.151419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.151356 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerDied","Data":"ab2d64dab1b173675a5cebb5639e62cb7de7666eeeab311ecfbb0804bb0f6cc4"} Nov 03 19:45:03.151682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.151665 5233 scope.go:122] "RemoveContainer" containerID="ab2d64dab1b173675a5cebb5639e62cb7de7666eeeab311ecfbb0804bb0f6cc4" Nov 03 19:45:03.180622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.180589 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpwl\" (UniqueName: \"kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl\") pod \"payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.180753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.180730 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.182089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.182067 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.199708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.199677 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpwl\" (UniqueName: \"kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl\") pod \"payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.405419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.405389 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:03.782222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:03.782195 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4"] Nov 03 19:45:03.789596 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:03.789562 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1276938_c683_42a6_9ea9_35fe93da8e7a.slice/crio-56fee93b760953a393078b2d95683e61e2013a02d2cbd158f737a6a172732bfa WatchSource:0}: Error finding container 56fee93b760953a393078b2d95683e61e2013a02d2cbd158f737a6a172732bfa: Status 404 returned error can't find the container with id 56fee93b760953a393078b2d95683e61e2013a02d2cbd158f737a6a172732bfa Nov 03 19:45:04.032708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.032670 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k"] Nov 03 19:45:04.044459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.044422 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c704263a-qsm8k"] Nov 03 19:45:04.165686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.165596 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" event={"ID":"d1276938-c683-42a6-9ea9-35fe93da8e7a","Type":"ContainerStarted","Data":"987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1"} Nov 03 19:45:04.165686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.165654 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" event={"ID":"d1276938-c683-42a6-9ea9-35fe93da8e7a","Type":"ContainerStarted","Data":"56fee93b760953a393078b2d95683e61e2013a02d2cbd158f737a6a172732bfa"} Nov 03 19:45:04.166155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.165687 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:04.168039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.168004 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerStarted","Data":"fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c"} Nov 03 19:45:04.170280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.170251 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerStarted","Data":"5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0"} Nov 03 19:45:04.172275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.172247 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" event={"ID":"052ebdb6-5f1a-4923-ad1d-98a1150991d7","Type":"ContainerStarted","Data":"ca473ee73baf9e363a1a35d0b45343ef23c27eef5e16d126927197034ad7abdd"} Nov 03 19:45:04.174355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.174330 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" event={"ID":"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8","Type":"ContainerStarted","Data":"dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8"} Nov 03 19:45:04.176424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.176404 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerStarted","Data":"7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473"} Nov 03 19:45:04.178496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.178475 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerStarted","Data":"0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f"} Nov 03 19:45:04.196895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.196854 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" podStartSLOduration=2.196841404 podStartE2EDuration="2.196841404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.189390459 +0000 UTC m=+1070.530503944" watchObservedRunningTime="2026-11-03 19:45:04.196841404 +0000 UTC m=+1070.537954928" Nov 03 19:45:04.346660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:04.346595 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" podStartSLOduration=4.346574384 podStartE2EDuration="4.346574384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.345110311 +0000 UTC m=+1070.686223796" watchObservedRunningTime="2026-11-03 19:45:04.346574384 +0000 UTC m=+1070.687687871" Nov 03 19:45:04.526323 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:04.526293 5233 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fee47e5_5bd6_42ee_9517_1461bf6e8764.slice/crio-conmon-5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:45:05.192776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.192744 5233 generic.go:356] "Generic (PLEG): container finished" podID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerID="dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8" exitCode=60 Nov 03 19:45:05.193293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.192834 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" event={"ID":"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8","Type":"ContainerDied","Data":"dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8"} Nov 03 19:45:05.193293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.192934 5233 scope.go:122] "RemoveContainer" containerID="608b03ac00d178e86b7038ac3bf4bef6c8f4080e2e2d7919148091bd6342cabb" Nov 03 19:45:05.193293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.193289 5233 scope.go:122] "RemoveContainer" containerID="dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8" Nov 03 19:45:05.193771 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:05.193746 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c_odh-ai-gateway-infra(37cb8835-75ed-4d1c-9c2d-7e21a8e671b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" Nov 03 19:45:05.195295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.195269 5233 generic.go:356] "Generic (PLEG): container finished" podID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerID="7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473" exitCode=60 Nov 03 19:45:05.195382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.195355 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerDied","Data":"7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473"} Nov 03 19:45:05.195642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.195625 5233 scope.go:122] "RemoveContainer" containerID="7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473" Nov 03 19:45:05.196045 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:05.196024 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx_odh-ai-gateway-infra(d03e57be-f613-4e5c-8e41-8b9cca4b82a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" Nov 03 19:45:05.197631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.197612 5233 generic.go:356] "Generic (PLEG): container finished" podID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerID="0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f" exitCode=60 Nov 03 19:45:05.197712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.197695 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerDied","Data":"0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f"} Nov 03 19:45:05.197909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.197892 5233 scope.go:122] "RemoveContainer" containerID="0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f" Nov 03 19:45:05.198317 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:05.198280 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh_odh-ai-gateway-infra(2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" Nov 03 19:45:05.200066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.200044 5233 generic.go:356] "Generic (PLEG): container finished" podID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerID="fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c" exitCode=60 Nov 03 19:45:05.200158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.200117 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerDied","Data":"fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c"} Nov 03 19:45:05.200379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.200357 5233 scope.go:122] "RemoveContainer" containerID="fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c" Nov 03 19:45:05.200687 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:05.200660 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x_odh-ai-gateway-infra(f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" Nov 03 19:45:05.202447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.202426 5233 generic.go:356] "Generic (PLEG): container finished" podID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerID="5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0" exitCode=60 Nov 03 19:45:05.202567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.202522 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerDied","Data":"5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0"} Nov 03 19:45:05.202717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.202703 5233 scope.go:122] "RemoveContainer" containerID="5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0" Nov 03 19:45:05.202990 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:05.202956 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-zkssn_odh-ai-gateway-infra(6fee47e5-5bd6-42ee-9517-1461bf6e8764)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" Nov 03 19:45:05.204694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.204669 5233 generic.go:356] "Generic (PLEG): container finished" podID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerID="ca473ee73baf9e363a1a35d0b45343ef23c27eef5e16d126927197034ad7abdd" exitCode=60 Nov 03 19:45:05.204783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.204710 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" event={"ID":"052ebdb6-5f1a-4923-ad1d-98a1150991d7","Type":"ContainerDied","Data":"ca473ee73baf9e363a1a35d0b45343ef23c27eef5e16d126927197034ad7abdd"} Nov 03 19:45:05.205107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.205088 5233 scope.go:122] "RemoveContainer" containerID="ca473ee73baf9e363a1a35d0b45343ef23c27eef5e16d126927197034ad7abdd" Nov 03 19:45:05.205412 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:05.205393 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx_odh-ai-gateway-infra(052ebdb6-5f1a-4923-ad1d-98a1150991d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" Nov 03 19:45:05.209371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.209350 5233 scope.go:122] "RemoveContainer" containerID="fffe67a2157b04397165cde828dd7da4abc7fa5825cbff4be72364899a47b333" Nov 03 19:45:05.225461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.225444 5233 scope.go:122] "RemoveContainer" containerID="c857920a23c039169c0c447f032b018b004562163dddc10690c986515c377107" Nov 03 19:45:05.241845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.241827 5233 scope.go:122] "RemoveContainer" containerID="ab2d64dab1b173675a5cebb5639e62cb7de7666eeeab311ecfbb0804bb0f6cc4" Nov 03 19:45:05.266136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.266113 5233 scope.go:122] "RemoveContainer" containerID="d0d19b26bee2552da0bc6b3e57702851c1bec083356e03a4420b527d9d5819ef" Nov 03 19:45:05.282767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.282743 5233 scope.go:122] "RemoveContainer" containerID="23345ff0669c1c196a03267a8bbae4200dab8f8529e1e0fd7e99eae8cf9dacc9" Nov 03 19:45:05.368003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.367954 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx"] Nov 03 19:45:05.844407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:05.844368 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44e9a18-c94b-4057-bf5e-e67b9cca9af9" path="/var/lib/kubelet/pods/b44e9a18-c94b-4057-bf5e-e67b9cca9af9/volumes" Nov 03 19:45:06.002917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.002885 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7"] Nov 03 19:45:06.013631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.013595 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14a79569-h97n7"] Nov 03 19:45:06.241288 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.241262 5233 scope.go:122] "RemoveContainer" containerID="fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c" Nov 03 19:45:06.241724 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:06.241634 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x_odh-ai-gateway-infra(f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" Nov 03 19:45:06.243284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.243264 5233 scope.go:122] "RemoveContainer" containerID="5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0" Nov 03 19:45:06.243565 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:06.243543 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-zkssn_odh-ai-gateway-infra(6fee47e5-5bd6-42ee-9517-1461bf6e8764)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" Nov 03 19:45:06.244872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.244852 5233 scope.go:122] "RemoveContainer" containerID="ca473ee73baf9e363a1a35d0b45343ef23c27eef5e16d126927197034ad7abdd" Nov 03 19:45:06.245119 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:06.245099 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx_odh-ai-gateway-infra(052ebdb6-5f1a-4923-ad1d-98a1150991d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" Nov 03 19:45:06.247079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.247052 5233 scope.go:122] "RemoveContainer" containerID="dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8" Nov 03 19:45:06.247395 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:06.247374 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c_odh-ai-gateway-infra(37cb8835-75ed-4d1c-9c2d-7e21a8e671b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" Nov 03 19:45:06.248525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.248508 5233 scope.go:122] "RemoveContainer" containerID="7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473" Nov 03 19:45:06.248744 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:06.248724 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx_odh-ai-gateway-infra(d03e57be-f613-4e5c-8e41-8b9cca4b82a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" Nov 03 19:45:06.250163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.250145 5233 scope.go:122] "RemoveContainer" containerID="0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f" Nov 03 19:45:06.250383 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:06.250365 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh_odh-ai-gateway-infra(2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" Nov 03 19:45:06.297158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.297108 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" podUID="0eb3051b-683d-4c2c-8e64-4e8a7411f60a" containerName="payload-pre-processing" containerID="cri-o://21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a" gracePeriod=25 Nov 03 19:45:06.297333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.297119 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" podUID="065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" containerName="payload-processing" containerID="cri-o://75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774" gracePeriod=25 Nov 03 19:45:06.638609 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.638576 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:45:06.676086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.676057 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:45:06.725099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.725063 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume\") pod \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " Nov 03 19:45:06.725409 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.725386 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume\") pod \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " Nov 03 19:45:06.725512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.725428 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q\" (UniqueName: \"kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q\") pod \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\" (UID: \"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981\") " Nov 03 19:45:06.725512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.725488 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f\" (UniqueName: \"kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f\") pod \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\" (UID: \"0eb3051b-683d-4c2c-8e64-4e8a7411f60a\") " Nov 03 19:45:06.725634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.725497 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume" pod "0eb3051b-683d-4c2c-8e64-4e8a7411f60a" (UID: "0eb3051b-683d-4c2c-8e64-4e8a7411f60a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:06.725821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.725794 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume" pod "065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" (UID: "065fb1d8-cbf5-4c6d-b29f-62bab5fc1981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:06.726215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.726189 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:06.726306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.726217 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:06.728711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.728684 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f" pod "0eb3051b-683d-4c2c-8e64-4e8a7411f60a" (UID: "0eb3051b-683d-4c2c-8e64-4e8a7411f60a"). InnerVolumeSpecName "kube-api-access-hps6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:06.733153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.733125 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q" pod "065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" (UID: "065fb1d8-cbf5-4c6d-b29f-62bab5fc1981"). InnerVolumeSpecName "kube-api-access-wb68q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:06.828258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.828165 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb68q\" (UniqueName: \"kubernetes.io/projected/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981-kube-api-access-wb68q\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:06.828258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:06.828195 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hps6f\" (UniqueName: \"kubernetes.io/projected/0eb3051b-683d-4c2c-8e64-4e8a7411f60a-kube-api-access-hps6f\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:07.263234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.263198 5233 generic.go:356] "Generic (PLEG): container finished" podID="065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" containerID="75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774" exitCode=0 Nov 03 19:45:07.263805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.263249 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" Nov 03 19:45:07.263805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.263286 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" event={"ID":"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981","Type":"ContainerDied","Data":"75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774"} Nov 03 19:45:07.263805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.263352 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj" event={"ID":"065fb1d8-cbf5-4c6d-b29f-62bab5fc1981","Type":"ContainerDied","Data":"50ac7fcf407ddf7cf2a0fb089ea8efa1cc0dfd11f93e3b31ec3b08c77d7676f2"} Nov 03 19:45:07.263805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.263386 5233 scope.go:122] "RemoveContainer" containerID="75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774" Nov 03 19:45:07.269952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.269929 5233 generic.go:356] "Generic (PLEG): container finished" podID="0eb3051b-683d-4c2c-8e64-4e8a7411f60a" containerID="21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a" exitCode=0 Nov 03 19:45:07.270061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.269957 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" event={"ID":"0eb3051b-683d-4c2c-8e64-4e8a7411f60a","Type":"ContainerDied","Data":"21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a"} Nov 03 19:45:07.270061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.270042 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" event={"ID":"0eb3051b-683d-4c2c-8e64-4e8a7411f60a","Type":"ContainerDied","Data":"46145dbbaeab00b3fc4094aeb355c8f8a4ca0c00e2dd3e6b345f5276a1eeb464"} Nov 03 19:45:07.270461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.270372 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8" Nov 03 19:45:07.284529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.284508 5233 scope.go:122] "RemoveContainer" containerID="75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774" Nov 03 19:45:07.284829 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:07.284808 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774\": container with ID starting with 75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774 not found: ID does not exist" containerID="75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774" Nov 03 19:45:07.284884 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.284837 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774"} err="failed to get container status \"75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774\": rpc error: code = NotFound desc = could not find container \"75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774\": container with ID starting with 75cc8723d0113bbbd8702c6fa3bafa62fb93ef6dfa4ed114d290b791a6acc774 not found: ID does not exist" Nov 03 19:45:07.284884 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.284853 5233 scope.go:122] "RemoveContainer" containerID="21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a" Nov 03 19:45:07.307795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.307756 5233 scope.go:122] "RemoveContainer" containerID="21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a" Nov 03 19:45:07.308604 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:07.308569 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a\": container with ID starting with 21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a not found: ID does not exist" containerID="21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a" Nov 03 19:45:07.308822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.308615 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a"} err="failed to get container status \"21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a\": rpc error: code = NotFound desc = could not find container \"21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a\": container with ID starting with 21c3bdd2aaeeb3240312d94a0ff0f0bf282394b941730211f2ad3b1c7671000a not found: ID does not exist" Nov 03 19:45:07.317553 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.317517 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8"] Nov 03 19:45:07.326077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.326052 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a79569-579f478bc-2qqv8"] Nov 03 19:45:07.334051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.334028 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj"] Nov 03 19:45:07.341349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.341324 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a79569-7856768d9f-854tj"] Nov 03 19:45:07.479413 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.479389 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:07.542040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.542007 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf\" (UniqueName: \"kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf\") pod \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " Nov 03 19:45:07.542248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.542215 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca\") pod \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\" (UID: \"052ebdb6-5f1a-4923-ad1d-98a1150991d7\") " Nov 03 19:45:07.542629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.542597 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca" pod "052ebdb6-5f1a-4923-ad1d-98a1150991d7" (UID: "052ebdb6-5f1a-4923-ad1d-98a1150991d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:07.543207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.543180 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052ebdb6-5f1a-4923-ad1d-98a1150991d7-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:07.544459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.544432 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf" pod "052ebdb6-5f1a-4923-ad1d-98a1150991d7" (UID: "052ebdb6-5f1a-4923-ad1d-98a1150991d7"). InnerVolumeSpecName "kube-api-access-lnbvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:07.644535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.644446 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnbvf\" (UniqueName: \"kubernetes.io/projected/052ebdb6-5f1a-4923-ad1d-98a1150991d7-kube-api-access-lnbvf\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:07.842897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.842861 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" path="/var/lib/kubelet/pods/065fb1d8-cbf5-4c6d-b29f-62bab5fc1981/volumes" Nov 03 19:45:07.843822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.843798 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb3051b-683d-4c2c-8e64-4e8a7411f60a" path="/var/lib/kubelet/pods/0eb3051b-683d-4c2c-8e64-4e8a7411f60a/volumes" Nov 03 19:45:07.844587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:07.844560 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6974d641-034c-425a-bfea-d6a88f35a552" path="/var/lib/kubelet/pods/6974d641-034c-425a-bfea-d6a88f35a552/volumes" Nov 03 19:45:08.283034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:08.282937 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" event={"ID":"052ebdb6-5f1a-4923-ad1d-98a1150991d7","Type":"ContainerDied","Data":"529e214345bfec811f8f74dcc0c4345e0f4cf56e72accb4e0e07b0ea9666310f"} Nov 03 19:45:08.283034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:08.282956 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx" Nov 03 19:45:08.283034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:08.283018 5233 scope.go:122] "RemoveContainer" containerID="ca473ee73baf9e363a1a35d0b45343ef23c27eef5e16d126927197034ad7abdd" Nov 03 19:45:08.306655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:08.306628 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx"] Nov 03 19:45:08.313298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:08.313273 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-14a79569-29895585-qnwgx"] Nov 03 19:45:09.062918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:09.062884 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp"] Nov 03 19:45:09.070441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:09.070412 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84a6af3-6h4jp"] Nov 03 19:45:09.841932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:09.841900 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" path="/var/lib/kubelet/pods/052ebdb6-5f1a-4923-ad1d-98a1150991d7/volumes" Nov 03 19:45:09.842518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:09.842491 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862dd125-578c-4cac-a4a8-cbb2f0d3d4e8" path="/var/lib/kubelet/pods/862dd125-578c-4cac-a4a8-cbb2f0d3d4e8/volumes" Nov 03 19:45:10.039043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:10.035048 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg"] Nov 03 19:45:10.042101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:10.042065 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg"] Nov 03 19:45:10.052071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:10.052042 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82262dac-b5lrg"] Nov 03 19:45:10.056343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:10.056319 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a1f30d-zjcrg"] Nov 03 19:45:11.027246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:11.027207 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx"] Nov 03 19:45:11.033830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:11.033804 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6236d392-7vdcx"] Nov 03 19:45:11.840868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:11.840831 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3254f83c-ba24-4252-a932-46ee800ccd82" path="/var/lib/kubelet/pods/3254f83c-ba24-4252-a932-46ee800ccd82/volumes" Nov 03 19:45:11.841557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:11.841529 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8e1af4-cac2-4d73-b0d5-8de6357cba5e" path="/var/lib/kubelet/pods/8b8e1af4-cac2-4d73-b0d5-8de6357cba5e/volumes" Nov 03 19:45:11.842202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:11.842185 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15" path="/var/lib/kubelet/pods/dd762f9f-a9dc-4fa1-b1bf-f7973ab5ff15/volumes" Nov 03 19:45:14.556300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.556266 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp"] Nov 03 19:45:14.557140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557117 5233 cpu_manager.go:414] "containerMap: removing container" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerName="cleanup" Nov 03 19:45:14.557140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557137 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerName="cleanup" Nov 03 19:45:14.557344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557178 5233 cpu_manager.go:414] "containerMap: removing container" podUID="065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" containerName="payload-processing" Nov 03 19:45:14.557344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557186 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" containerName="payload-processing" Nov 03 19:45:14.557344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557197 5233 cpu_manager.go:414] "containerMap: removing container" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerName="cleanup" Nov 03 19:45:14.557344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557206 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerName="cleanup" Nov 03 19:45:14.557344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557289 5233 cpu_manager.go:414] "containerMap: removing container" podUID="0eb3051b-683d-4c2c-8e64-4e8a7411f60a" containerName="payload-pre-processing" Nov 03 19:45:14.557344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557297 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb3051b-683d-4c2c-8e64-4e8a7411f60a" containerName="payload-pre-processing" Nov 03 19:45:14.557629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557520 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb3051b-683d-4c2c-8e64-4e8a7411f60a" containerName="payload-pre-processing" Nov 03 19:45:14.557629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557539 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerName="cleanup" Nov 03 19:45:14.557629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557550 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="065fb1d8-cbf5-4c6d-b29f-62bab5fc1981" containerName="payload-processing" Nov 03 19:45:14.557629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.557560 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="052ebdb6-5f1a-4923-ad1d-98a1150991d7" containerName="cleanup" Nov 03 19:45:14.563444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.563423 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.574912 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.573516 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-763a69\"" Nov 03 19:45:14.576873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.576849 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp"] Nov 03 19:45:14.691407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.690557 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch6w\" (UniqueName: \"kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w\") pod \"payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.691407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.690641 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.795998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.795455 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch6w\" (UniqueName: \"kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w\") pod \"payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.795998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.795534 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.816049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.796739 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.820265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.819047 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch6w\" (UniqueName: \"kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w\") pod \"payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:14.846996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.840862 5233 scope.go:122] "RemoveContainer" containerID="0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f" Nov 03 19:45:14.866009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.851260 5233 scope.go:122] "RemoveContainer" containerID="dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8" Nov 03 19:45:14.900718 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:14.898352 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:15.207911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:15.207887 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:45:15.304483 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:15.304436 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8705ab5_c573_4712_97f9_4bc7ec7d97fc.slice/crio-fd59dd4270654c19ceaad55d9077a47db1fb61e332153f47e651ee76fde25806 WatchSource:0}: Error finding container fd59dd4270654c19ceaad55d9077a47db1fb61e332153f47e651ee76fde25806: Status 404 returned error can't find the container with id fd59dd4270654c19ceaad55d9077a47db1fb61e332153f47e651ee76fde25806 Nov 03 19:45:15.306434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:15.306412 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp"] Nov 03 19:45:15.529085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:15.526819 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerStarted","Data":"31759b29785fa9ee8faa38b7b130fca7b9d8d55b6d7beda6b371895c259e2f55"} Nov 03 19:45:15.533315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:15.533282 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" event={"ID":"a8705ab5-c573-4712-97f9-4bc7ec7d97fc","Type":"ContainerStarted","Data":"fd59dd4270654c19ceaad55d9077a47db1fb61e332153f47e651ee76fde25806"} Nov 03 19:45:16.548026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.547954 5233 generic.go:356] "Generic (PLEG): container finished" podID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerID="02e7d3d06804764e14957d06b019e6f06a843a9df901cc9d8fb7d07e72e0f50d" exitCode=60 Nov 03 19:45:16.548026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.547998 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" event={"ID":"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8","Type":"ContainerDied","Data":"02e7d3d06804764e14957d06b019e6f06a843a9df901cc9d8fb7d07e72e0f50d"} Nov 03 19:45:16.548549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.548054 5233 scope.go:122] "RemoveContainer" containerID="dfd8f466e2af11ad15026941e97fa2d8503efe1a8607b03ef389c60b849890a8" Nov 03 19:45:16.548549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.548506 5233 scope.go:122] "RemoveContainer" containerID="02e7d3d06804764e14957d06b019e6f06a843a9df901cc9d8fb7d07e72e0f50d" Nov 03 19:45:16.548923 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:16.548899 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c_odh-ai-gateway-infra(37cb8835-75ed-4d1c-9c2d-7e21a8e671b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" Nov 03 19:45:16.550255 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.550236 5233 generic.go:356] "Generic (PLEG): container finished" podID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerID="31759b29785fa9ee8faa38b7b130fca7b9d8d55b6d7beda6b371895c259e2f55" exitCode=60 Nov 03 19:45:16.550347 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.550295 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerDied","Data":"31759b29785fa9ee8faa38b7b130fca7b9d8d55b6d7beda6b371895c259e2f55"} Nov 03 19:45:16.550655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.550632 5233 scope.go:122] "RemoveContainer" containerID="31759b29785fa9ee8faa38b7b130fca7b9d8d55b6d7beda6b371895c259e2f55" Nov 03 19:45:16.551023 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:16.551000 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh_odh-ai-gateway-infra(2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" Nov 03 19:45:16.552029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.552003 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" event={"ID":"a8705ab5-c573-4712-97f9-4bc7ec7d97fc","Type":"ContainerStarted","Data":"48dc654c47f2a21b731214a25cbf139c9b1aea9b8e99fc4e068437466cdb3bac"} Nov 03 19:45:16.552137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.552121 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:16.563598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.563580 5233 scope.go:122] "RemoveContainer" containerID="0a9b902cb4612e7f451500788abe8c52de061c062d89996cb94ad8a231f06d9f" Nov 03 19:45:16.584873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.584850 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh"] Nov 03 19:45:16.635036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.629317 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" podStartSLOduration=2.629298663 podStartE2EDuration="2.629298663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:16.624373517 +0000 UTC m=+1082.965487004" watchObservedRunningTime="2026-11-03 19:45:16.629298663 +0000 UTC m=+1082.970412148" Nov 03 19:45:16.829529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:16.829426 5233 scope.go:122] "RemoveContainer" containerID="7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473" Nov 03 19:45:17.566759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:17.566722 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerStarted","Data":"eda52c86c731782ba85e45611404a34d22da8df202cca3208a0d632289a2cd68"} Nov 03 19:45:17.584842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:17.584804 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c"] Nov 03 19:45:18.159115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.159075 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9"] Nov 03 19:45:18.167514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.166375 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e8e24c3-nnlq9"] Nov 03 19:45:18.296678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.296650 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:18.298687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.298668 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:18.434272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.434239 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca\") pod \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " Nov 03 19:45:18.434472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.434365 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca\") pod \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " Nov 03 19:45:18.434472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.434432 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx\" (UniqueName: \"kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx\") pod \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\" (UID: \"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76\") " Nov 03 19:45:18.434472 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.434465 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854\" (UniqueName: \"kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854\") pod \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\" (UID: \"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8\") " Nov 03 19:45:18.434873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.434846 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca" pod "37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" (UID: "37cb8835-75ed-4d1c-9c2d-7e21a8e671b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:18.435084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.435060 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca" pod "2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" (UID: "2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:18.435451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.435424 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:18.435544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.435455 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:18.436793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.436769 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx" pod "2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" (UID: "2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76"). InnerVolumeSpecName "kube-api-access-wktpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:18.437124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.437104 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854" pod "37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" (UID: "37cb8835-75ed-4d1c-9c2d-7e21a8e671b8"). InnerVolumeSpecName "kube-api-access-9x854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:18.536944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.536851 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wktpx\" (UniqueName: \"kubernetes.io/projected/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76-kube-api-access-wktpx\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:18.536944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.536893 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x854\" (UniqueName: \"kubernetes.io/projected/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8-kube-api-access-9x854\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:18.580820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.580791 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" Nov 03 19:45:18.580820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.580802 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c" event={"ID":"37cb8835-75ed-4d1c-9c2d-7e21a8e671b8","Type":"ContainerDied","Data":"a27b0ccb45d1c0844025cc0ee3ef06f997dba57838c3f048651b6b581ed8c128"} Nov 03 19:45:18.581358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.580897 5233 scope.go:122] "RemoveContainer" containerID="02e7d3d06804764e14957d06b019e6f06a843a9df901cc9d8fb7d07e72e0f50d" Nov 03 19:45:18.589403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.589376 5233 generic.go:356] "Generic (PLEG): container finished" podID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerID="eda52c86c731782ba85e45611404a34d22da8df202cca3208a0d632289a2cd68" exitCode=60 Nov 03 19:45:18.589542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.589497 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerDied","Data":"eda52c86c731782ba85e45611404a34d22da8df202cca3208a0d632289a2cd68"} Nov 03 19:45:18.590691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.589935 5233 scope.go:122] "RemoveContainer" containerID="eda52c86c731782ba85e45611404a34d22da8df202cca3208a0d632289a2cd68" Nov 03 19:45:18.590691 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:18.590389 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx_odh-ai-gateway-infra(d03e57be-f613-4e5c-8e41-8b9cca4b82a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" Nov 03 19:45:18.606057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.602321 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" Nov 03 19:45:18.606057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.602318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh" event={"ID":"2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76","Type":"ContainerDied","Data":"942fed3709b710be2cadda3a833cbde171297679da610cdf73041211edb873b8"} Nov 03 19:45:18.609410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.608876 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx"] Nov 03 19:45:18.612048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.612027 5233 scope.go:122] "RemoveContainer" containerID="7b88c17f9d507554436bbba750cf49b76ce2ef960f0053d77e65c476ed289473" Nov 03 19:45:18.642131 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.641921 5233 scope.go:122] "RemoveContainer" containerID="31759b29785fa9ee8faa38b7b130fca7b9d8d55b6d7beda6b371895c259e2f55" Nov 03 19:45:18.655745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.655717 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c"] Nov 03 19:45:18.662900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.662874 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2c3fc4-29895585-tlj2c"] Nov 03 19:45:18.669649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.669627 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh"] Nov 03 19:45:18.674931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.674905 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-435bd8-29895585-js2wh"] Nov 03 19:45:18.829606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.829521 5233 scope.go:122] "RemoveContainer" containerID="fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c" Nov 03 19:45:18.829925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:18.829898 5233 scope.go:122] "RemoveContainer" containerID="5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0" Nov 03 19:45:19.620898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.620860 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerStarted","Data":"c539fdd7475d8f76b0650eb52a54986b8301e7310337a740acbc2f70dacf0f34"} Nov 03 19:45:19.623494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.623470 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerStarted","Data":"29012b13e649d4f934a706859706593af7faef3fb779cc4ad33f5c512470764e"} Nov 03 19:45:19.832901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.832873 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:19.841089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.841063 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" path="/var/lib/kubelet/pods/2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76/volumes" Nov 03 19:45:19.841659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.841643 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" path="/var/lib/kubelet/pods/37cb8835-75ed-4d1c-9c2d-7e21a8e671b8/volumes" Nov 03 19:45:19.842196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.842181 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d45166-f204-4f00-95c6-f97de55c5ac4" path="/var/lib/kubelet/pods/a1d45166-f204-4f00-95c6-f97de55c5ac4/volumes" Nov 03 19:45:19.963320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.963298 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k\" (UniqueName: \"kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k\") pod \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " Nov 03 19:45:19.963469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.963453 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca\") pod \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\" (UID: \"d03e57be-f613-4e5c-8e41-8b9cca4b82a9\") " Nov 03 19:45:19.963804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.963779 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca" pod "d03e57be-f613-4e5c-8e41-8b9cca4b82a9" (UID: "d03e57be-f613-4e5c-8e41-8b9cca4b82a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:19.964194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.964175 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:19.965411 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:19.965387 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k" pod "d03e57be-f613-4e5c-8e41-8b9cca4b82a9" (UID: "d03e57be-f613-4e5c-8e41-8b9cca4b82a9"). InnerVolumeSpecName "kube-api-access-lwx8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:20.028774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.028714 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx"] Nov 03 19:45:20.035923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.035892 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67f5d9-sjhxx"] Nov 03 19:45:20.065773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.065737 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwx8k\" (UniqueName: \"kubernetes.io/projected/d03e57be-f613-4e5c-8e41-8b9cca4b82a9-kube-api-access-lwx8k\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:20.644032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.643960 5233 generic.go:356] "Generic (PLEG): container finished" podID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerID="c539fdd7475d8f76b0650eb52a54986b8301e7310337a740acbc2f70dacf0f34" exitCode=60 Nov 03 19:45:20.644457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.644037 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerDied","Data":"c539fdd7475d8f76b0650eb52a54986b8301e7310337a740acbc2f70dacf0f34"} Nov 03 19:45:20.644457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.644116 5233 scope.go:122] "RemoveContainer" containerID="fc7e18c1bf1e726d3a1f671e7c82f29dc070c1ca3db339cd42661484a84a792c" Nov 03 19:45:20.644578 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.644521 5233 scope.go:122] "RemoveContainer" containerID="c539fdd7475d8f76b0650eb52a54986b8301e7310337a740acbc2f70dacf0f34" Nov 03 19:45:20.644919 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:20.644896 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x_odh-ai-gateway-infra(f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" Nov 03 19:45:20.646422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.646402 5233 generic.go:356] "Generic (PLEG): container finished" podID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerID="29012b13e649d4f934a706859706593af7faef3fb779cc4ad33f5c512470764e" exitCode=60 Nov 03 19:45:20.646544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.646477 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerDied","Data":"29012b13e649d4f934a706859706593af7faef3fb779cc4ad33f5c512470764e"} Nov 03 19:45:20.646860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.646843 5233 scope.go:122] "RemoveContainer" containerID="29012b13e649d4f934a706859706593af7faef3fb779cc4ad33f5c512470764e" Nov 03 19:45:20.647198 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:20.647177 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-zkssn_odh-ai-gateway-infra(6fee47e5-5bd6-42ee-9517-1461bf6e8764)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" Nov 03 19:45:20.647840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.647811 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" event={"ID":"d03e57be-f613-4e5c-8e41-8b9cca4b82a9","Type":"ContainerDied","Data":"9188cdbffa4cc0e51d87eae9f124c7399ab16e005dc0c362457e002b7b3527bf"} Nov 03 19:45:20.647910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.647841 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx" Nov 03 19:45:20.657600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.657580 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x"] Nov 03 19:45:20.661768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.661751 5233 scope.go:122] "RemoveContainer" containerID="5ddd6301b6feba30ab15a6813e52a5a277ba7d41f585af7a9f46b0c6c44dc4b0" Nov 03 19:45:20.676757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.676741 5233 scope.go:122] "RemoveContainer" containerID="eda52c86c731782ba85e45611404a34d22da8df202cca3208a0d632289a2cd68" Nov 03 19:45:20.720187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.720156 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn"] Nov 03 19:45:20.730264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.730239 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx"] Nov 03 19:45:20.737353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:20.737332 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-05e217-29895585-5hzqx"] Nov 03 19:45:21.133078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:21.133034 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8"] Nov 03 19:45:21.144260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:21.144167 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4534906-45rw8"] Nov 03 19:45:21.290125 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:21.290084 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:45:21.909336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:21.903914 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a59d344-e0cb-4b55-95df-94d4e140192e" path="/var/lib/kubelet/pods/7a59d344-e0cb-4b55-95df-94d4e140192e/volumes" Nov 03 19:45:21.909336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:21.904687 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" path="/var/lib/kubelet/pods/d03e57be-f613-4e5c-8e41-8b9cca4b82a9/volumes" Nov 03 19:45:21.909336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:21.905680 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69293c2-5614-4aec-9d76-c589f1ff7c81" path="/var/lib/kubelet/pods/e69293c2-5614-4aec-9d76-c589f1ff7c81/volumes" Nov 03 19:45:22.045210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.045173 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt"] Nov 03 19:45:22.060103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.060074 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f7c26f-5ghdt"] Nov 03 19:45:22.184660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.183871 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:22.188993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.186893 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:22.264708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.262862 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca\") pod \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " Nov 03 19:45:22.264708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.263126 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g\" (UniqueName: \"kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g\") pod \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " Nov 03 19:45:22.264708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.263231 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt\" (UniqueName: \"kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt\") pod \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\" (UID: \"6fee47e5-5bd6-42ee-9517-1461bf6e8764\") " Nov 03 19:45:22.264708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.263284 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca\") pod \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\" (UID: \"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8\") " Nov 03 19:45:22.267228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.267198 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca" pod "6fee47e5-5bd6-42ee-9517-1461bf6e8764" (UID: "6fee47e5-5bd6-42ee-9517-1461bf6e8764"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:22.267857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.267787 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca" pod "f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" (UID: "f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:22.269077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.269034 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee47e5-5bd6-42ee-9517-1461bf6e8764-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:22.269077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.269063 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:22.270763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.270740 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt" pod "6fee47e5-5bd6-42ee-9517-1461bf6e8764" (UID: "6fee47e5-5bd6-42ee-9517-1461bf6e8764"). InnerVolumeSpecName "kube-api-access-htlmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:22.324358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.298195 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g" pod "f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" (UID: "f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8"). InnerVolumeSpecName "kube-api-access-f6h5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:22.393832 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.393744 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6h5g\" (UniqueName: \"kubernetes.io/projected/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8-kube-api-access-f6h5g\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:22.394016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.393858 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlmt\" (UniqueName: \"kubernetes.io/projected/6fee47e5-5bd6-42ee-9517-1461bf6e8764-kube-api-access-htlmt\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.501119 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s"] Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502677 5233 cpu_manager.go:414] "containerMap: removing container" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502693 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502706 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502712 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502721 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502727 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502740 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502744 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502751 5233 cpu_manager.go:414] "containerMap: removing container" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502756 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502772 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502776 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502790 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502796 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502824 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502828 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502838 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502843 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502853 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502857 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502866 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502870 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.502996 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503010 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503019 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503027 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503034 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503043 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503048 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503055 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503060 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503066 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:22.503993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.503073 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:22.516127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.510373 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.521005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.520965 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2b2e60-openshift-default-dockercfg-fwtzx\"" Nov 03 19:45:22.522359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.522330 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s"] Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.709992 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710125 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710176 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710334 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710360 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7bc\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710415 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710443 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710536 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.711052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.710564 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812516 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812555 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812598 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812666 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812695 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812815 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812843 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7bc\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812902 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.812928 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.813855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.813431 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.815423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.814876 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.815423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.815146 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.816956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.816117 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.816956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.816908 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.821526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.820586 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" event={"ID":"f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8","Type":"ContainerDied","Data":"7c3783b10c64c6107fd9a3781e8b85558ee05026a17113b9a67dc4950f19ec5f"} Nov 03 19:45:22.821526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.820669 5233 scope.go:122] "RemoveContainer" containerID="c539fdd7475d8f76b0650eb52a54986b8301e7310337a740acbc2f70dacf0f34" Nov 03 19:45:22.821526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.820880 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x" Nov 03 19:45:22.822697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.822660 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.833001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.831206 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" event={"ID":"6fee47e5-5bd6-42ee-9517-1461bf6e8764","Type":"ContainerDied","Data":"ba3f2fa89c68973e58b3417299c22cd55d68c94f4210c77457c9d49c6b03262a"} Nov 03 19:45:22.833001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.831311 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn" Nov 03 19:45:22.834223 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.833845 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.849777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.849358 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7bc\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.849777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.849518 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token\") pod \"e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:22.907183 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.907121 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x"] Nov 03 19:45:22.912070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.912033 5233 scope.go:122] "RemoveContainer" containerID="29012b13e649d4f934a706859706593af7faef3fb779cc4ad33f5c512470764e" Nov 03 19:45:22.931809 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.931775 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c5f6cb-29895585-znh7x"] Nov 03 19:45:22.942423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.942364 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn"] Nov 03 19:45:22.950109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:22.950079 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-zkssn"] Nov 03 19:45:23.050188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.050146 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj"] Nov 03 19:45:23.061160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.060770 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f2395a-fmhdj"] Nov 03 19:45:23.143689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.143655 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:23.465355 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:23.465324 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4b279e_a68c_4dc3_b33f_2da4afa5116d.slice/crio-9a662a5bda07d13826a01cdcbd86eaed9347729f222bd69ada1100f862e3f000 WatchSource:0}: Error finding container 9a662a5bda07d13826a01cdcbd86eaed9347729f222bd69ada1100f862e3f000: Status 404 returned error can't find the container with id 9a662a5bda07d13826a01cdcbd86eaed9347729f222bd69ada1100f862e3f000 Nov 03 19:45:23.468410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.468373 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:23.468522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.468443 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:23.468522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.468475 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:23.470586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.470561 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s"] Nov 03 19:45:23.851090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.851053 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b47b081-a69a-482b-9422-4201f9db2343" path="/var/lib/kubelet/pods/4b47b081-a69a-482b-9422-4201f9db2343/volumes" Nov 03 19:45:23.851662 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.851640 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" path="/var/lib/kubelet/pods/6fee47e5-5bd6-42ee-9517-1461bf6e8764/volumes" Nov 03 19:45:23.852259 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.852239 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05393ae-72bf-48b4-b871-c8c59d6e4301" path="/var/lib/kubelet/pods/a05393ae-72bf-48b4-b871-c8c59d6e4301/volumes" Nov 03 19:45:23.853197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.853176 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8" path="/var/lib/kubelet/pods/f8cc15f2-4f17-4d23-a3f4-dae31e05a6b8/volumes" Nov 03 19:45:23.853715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.853692 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" event={"ID":"1a4b279e-a68c-4dc3-b33f-2da4afa5116d","Type":"ContainerStarted","Data":"0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b"} Nov 03 19:45:23.853806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.853732 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" event={"ID":"1a4b279e-a68c-4dc3-b33f-2da4afa5116d","Type":"ContainerStarted","Data":"9a662a5bda07d13826a01cdcbd86eaed9347729f222bd69ada1100f862e3f000"} Nov 03 19:45:23.882485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:23.882416 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podStartSLOduration=1.8823943760000001 podStartE2EDuration="1.882394376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:23.878571761 +0000 UTC m=+1090.219685247" watchObservedRunningTime="2026-11-03 19:45:23.882394376 +0000 UTC m=+1090.223507862" Nov 03 19:45:24.142539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:24.142440 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p"] Nov 03 19:45:24.165065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:24.154511 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:24.174775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:24.174744 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:24.175477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:24.175452 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec52661-22k5p"] Nov 03 19:45:24.866016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:24.865987 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:24.867120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:24.867095 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.617691 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm"] Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.618856 5233 cpu_manager.go:414] "containerMap: removing container" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.618877 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.618894 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.618901 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.618930 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.618940 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.619033 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.619044 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.619246 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fee47e5-5bd6-42ee-9517-1461bf6e8764" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.619258 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03e57be-f613-4e5c-8e41-8b9cca4b82a9" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.619270 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8b1ba6-0b40-4ed2-a47e-dfa99a32aa76" containerName="cleanup" Nov 03 19:45:25.620031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.619282 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cb8835-75ed-4d1c-9c2d-7e21a8e671b8" containerName="cleanup" Nov 03 19:45:25.625427 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.625404 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.633378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.630229 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:45:25.633378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.630895 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:45:25.633378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.630952 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm"] Nov 03 19:45:25.789051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.789005 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.789323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.789293 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpbh\" (UniqueName: \"kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.789480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.789451 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.839264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.839227 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48a00f8-5c3d-463b-b3fb-a7ff07fcca04" path="/var/lib/kubelet/pods/e48a00f8-5c3d-463b-b3fb-a7ff07fcca04/volumes" Nov 03 19:45:25.890737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.890655 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpbh\" (UniqueName: \"kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.890737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.890734 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.891026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.890868 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.891494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.891470 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.893298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.893273 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.907774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.907752 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpbh\" (UniqueName: \"kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh\") pod \"maas-api-revoke-sub-c43673be-vkvlm\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:25.953715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:25.953686 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:26.259951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:26.257107 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm"] Nov 03 19:45:26.261806 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:45:26.261765 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f48abdb_9005_43b1_830d_27fbc0b3f661.slice/crio-cd2e860314640ab318a9e72266340acb27586602514afa9f183886d2d441d242 WatchSource:0}: Error finding container cd2e860314640ab318a9e72266340acb27586602514afa9f183886d2d441d242: Status 404 returned error can't find the container with id cd2e860314640ab318a9e72266340acb27586602514afa9f183886d2d441d242 Nov 03 19:45:26.906114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:26.906072 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" event={"ID":"1f48abdb-9005-43b1-830d-27fbc0b3f661","Type":"ContainerStarted","Data":"2227682ca58334f14202439ae5a1f443065e237133b529ffe51e160b9c8dce2e"} Nov 03 19:45:26.906580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:26.906174 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" event={"ID":"1f48abdb-9005-43b1-830d-27fbc0b3f661","Type":"ContainerStarted","Data":"cd2e860314640ab318a9e72266340acb27586602514afa9f183886d2d441d242"} Nov 03 19:45:26.930127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:26.930079 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" podStartSLOduration=1.930065205 podStartE2EDuration="1.930065205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:26.925382848 +0000 UTC m=+1093.266496344" watchObservedRunningTime="2026-11-03 19:45:26.930065205 +0000 UTC m=+1093.271178686" Nov 03 19:45:27.571375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:27.571304 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:45:28.110518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:28.110144 5233 generic.go:356] "Generic (PLEG): container finished" podID="1f48abdb-9005-43b1-830d-27fbc0b3f661" containerID="2227682ca58334f14202439ae5a1f443065e237133b529ffe51e160b9c8dce2e" exitCode=0 Nov 03 19:45:28.110518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:28.110360 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" event={"ID":"1f48abdb-9005-43b1-830d-27fbc0b3f661","Type":"ContainerDied","Data":"2227682ca58334f14202439ae5a1f443065e237133b529ffe51e160b9c8dce2e"} Nov 03 19:45:29.296873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.296847 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:29.365204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.365172 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh\" (UniqueName: \"kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh\") pod \"1f48abdb-9005-43b1-830d-27fbc0b3f661\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " Nov 03 19:45:29.365407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.365280 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca\") pod \"1f48abdb-9005-43b1-830d-27fbc0b3f661\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " Nov 03 19:45:29.365407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.365334 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca\") pod \"1f48abdb-9005-43b1-830d-27fbc0b3f661\" (UID: \"1f48abdb-9005-43b1-830d-27fbc0b3f661\") " Nov 03 19:45:29.365736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.365693 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca" pod "1f48abdb-9005-43b1-830d-27fbc0b3f661" (UID: "1f48abdb-9005-43b1-830d-27fbc0b3f661"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:29.366193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.366160 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:29.368094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.368060 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh" pod "1f48abdb-9005-43b1-830d-27fbc0b3f661" (UID: "1f48abdb-9005-43b1-830d-27fbc0b3f661"). InnerVolumeSpecName "kube-api-access-nxpbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:29.368305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.368277 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca" pod "1f48abdb-9005-43b1-830d-27fbc0b3f661" (UID: "1f48abdb-9005-43b1-830d-27fbc0b3f661"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:45:29.467999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.467938 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpbh\" (UniqueName: \"kubernetes.io/projected/1f48abdb-9005-43b1-830d-27fbc0b3f661-kube-api-access-nxpbh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:29.467999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:29.467966 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f48abdb-9005-43b1-830d-27fbc0b3f661-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:30.197057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:30.196503 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" event={"ID":"1f48abdb-9005-43b1-830d-27fbc0b3f661","Type":"ContainerDied","Data":"cd2e860314640ab318a9e72266340acb27586602514afa9f183886d2d441d242"} Nov 03 19:45:30.197057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:30.196568 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2e860314640ab318a9e72266340acb27586602514afa9f183886d2d441d242" Nov 03 19:45:30.197057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:30.196638 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm" Nov 03 19:45:31.029007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:31.028953 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8"] Nov 03 19:45:31.037190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:31.037155 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3be2bdbe-vzlg8"] Nov 03 19:45:31.840112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:31.840077 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5dcbd5-ee40-4876-a3c6-4a699dd848e8" path="/var/lib/kubelet/pods/2e5dcbd5-ee40-4876-a3c6-4a699dd848e8/volumes" Nov 03 19:45:33.028689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:33.028650 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72"] Nov 03 19:45:33.034409 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:33.034381 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da62dd3-cfm72"] Nov 03 19:45:33.849001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:33.845768 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f726797a-52fd-44d8-b5a4-d9cd42446fd8" path="/var/lib/kubelet/pods/f726797a-52fd-44d8-b5a4-d9cd42446fd8/volumes" Nov 03 19:45:37.062988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:37.062192 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86"] Nov 03 19:45:37.073431 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:37.073399 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72c5758-pcs86"] Nov 03 19:45:37.838661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:37.838627 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39db2050-a067-48d8-89e3-5bb8b77f48d3" path="/var/lib/kubelet/pods/39db2050-a067-48d8-89e3-5bb8b77f48d3/volumes" Nov 03 19:45:39.286913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.286880 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv"] Nov 03 19:45:39.287706 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.287689 5233 cpu_manager.go:414] "containerMap: removing container" podUID="1f48abdb-9005-43b1-830d-27fbc0b3f661" containerName="revoke-keys" Nov 03 19:45:39.287776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.287707 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f48abdb-9005-43b1-830d-27fbc0b3f661" containerName="revoke-keys" Nov 03 19:45:39.287857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.287847 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f48abdb-9005-43b1-830d-27fbc0b3f661" containerName="revoke-keys" Nov 03 19:45:39.305269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.305228 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.319627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.319592 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv"] Nov 03 19:45:39.379882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.379849 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-64454a\"" Nov 03 19:45:39.389788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.389756 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjfd\" (UniqueName: \"kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd\") pod \"payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.389983 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.389948 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.504080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.504035 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjfd\" (UniqueName: \"kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd\") pod \"payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.507200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.507165 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.510136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.508250 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.564881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.564784 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjfd\" (UniqueName: \"kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd\") pod \"payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:39.633000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:39.627282 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:40.037631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.037235 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx"] Nov 03 19:45:40.047521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.045040 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024c683-xhwvx"] Nov 03 19:45:40.066342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.066315 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv"] Nov 03 19:45:40.362925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.362836 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" event={"ID":"15e58196-141a-4d6f-9815-034319ba770b","Type":"ContainerStarted","Data":"8db769c1d70764f201ef9015480704e743475ba3e71dfa0533cb7d4c9ff06864"} Nov 03 19:45:40.362925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.362893 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" event={"ID":"15e58196-141a-4d6f-9815-034319ba770b","Type":"ContainerStarted","Data":"ecd6437b203d000dbd2efa4b9b0aa288ba8f3b0053ff5d8705eda1a57c2c678d"} Nov 03 19:45:40.363363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.362933 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:40.385507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:40.385455 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" podStartSLOduration=1.385443089 podStartE2EDuration="1.385443089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:40.381424736 +0000 UTC m=+1106.722538223" watchObservedRunningTime="2026-11-03 19:45:40.385443089 +0000 UTC m=+1106.726556574" Nov 03 19:45:41.844363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:41.844324 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280a04f6-20e2-44e6-8ffe-7a8e98262ecb" path="/var/lib/kubelet/pods/280a04f6-20e2-44e6-8ffe-7a8e98262ecb/volumes" Nov 03 19:45:51.281362 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:45:51.281326 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:45:51.378895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:51.378870 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:45:52.167407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:52.167369 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs"] Nov 03 19:45:52.177650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:52.175184 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d38c760-ht9bs"] Nov 03 19:45:53.842441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:53.842367 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a162266-d513-4951-9a33-ddfadd231667" path="/var/lib/kubelet/pods/9a162266-d513-4951-9a33-ddfadd231667/volumes" Nov 03 19:45:57.040806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.040769 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd"] Nov 03 19:45:57.176545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.154117 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.227797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.218749 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eae28ff0-dockercfg-v5mjh\"" Nov 03 19:45:57.228353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.222504 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eae28ff0\"" Nov 03 19:45:57.278672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.270666 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.280531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.280501 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhvp\" (UniqueName: \"kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.400073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.397931 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.422343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.421281 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.423410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.422879 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhvp\" (UniqueName: \"kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.430093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.429203 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd"] Nov 03 19:45:57.527003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.524080 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhvp\" (UniqueName: \"kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.574518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.574481 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9"] Nov 03 19:45:57.619154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.618586 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:45:57.718930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.718069 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.798496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.749728 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9"] Nov 03 19:45:57.798496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.749848 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:57.874027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.861101 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:57.874027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.861428 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brml8\" (UniqueName: \"kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:57.918542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.913039 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.985440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.985321 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:57.986047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:57.986021 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brml8\" (UniqueName: \"kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:58.013032 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.000342 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:58.059149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.059106 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.099022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.093690 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brml8\" (UniqueName: \"kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:58.153997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.153198 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.292048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.291498 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:45:58.377073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.372061 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.826245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.826203 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:58.902621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.902587 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:45:58.906208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.903138 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.295756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.903170 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:45:59.434089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:58.925853 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.434089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.385312 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.434089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.341257 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.434089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.080237 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:45:59.437993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.434525 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:45:59.437993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.434849 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:45:59.437993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.434966 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:45:59.733228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.733165 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:45:59.927805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:45:59.923288 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.308555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:00.308503 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.380194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:00.380127 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.381875 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:00.381751 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.382320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:00.382278 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.386273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:00.386232 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.709200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:00.660463 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.014668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.009203 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.070941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.043100 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.207115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.207075 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.611996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.594728 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:46:01.611996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.595071 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:46:01.613161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.590695 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.640943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.640901 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Nov 03 19:46:01.690001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.664675 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.818494 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:46:01.818453 5233 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.694s" Nov 03 19:46:01.818910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.818886 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:46:01.819115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.819100 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:46:01.819255 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.819242 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:46:01.819384 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:01.819371 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:46:02.013769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.013633 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.074153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.071182 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:46:02.089207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.089156 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.096394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.096359 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:46:02.222154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.222091 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.613906 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.600259 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.624166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.616278 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:46:02.624166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.622042 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:46:02.743232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.743188 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd"] Nov 03 19:46:02.834560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.775163 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.834560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.788787 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:46:02.834560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.788873 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:46:02.839780 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:02.839735 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" event={"ID":"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8","Type":"ContainerStarted","Data":"83170f97f65ca0a24d7dbfef044b464c78c7210ef56d070e45688e3210a33af4"} Nov 03 19:46:03.052993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.052725 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 19:46:03.071383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.068190 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:46:03.080485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.080455 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9"] Nov 03 19:46:03.120557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.081341 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:46:03.120557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.090824 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:46:03.168996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.123065 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:03.181244 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.181216 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:46:03.220313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.194468 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" event={"ID":"06c01ec7-1fa5-43e1-8d49-facc35c28250","Type":"ContainerStarted","Data":"de6853ac274f559efad75341bab270efe1613069d81246329326ae5e022d1af1"} Nov 03 19:46:03.269525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.268397 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:46:03.283136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:03.283108 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:46:04.254429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:04.254060 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" event={"ID":"06c01ec7-1fa5-43e1-8d49-facc35c28250","Type":"ContainerStarted","Data":"95768a387f9a05aee8bc167b67c1b9598b101aef332960e3bf5e545db9c569ef"} Nov 03 19:46:04.254429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:04.254140 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:46:04.256026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:04.255998 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" event={"ID":"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8","Type":"ContainerStarted","Data":"1d5cbbd0978d42a3a31e0dee001163a839b4fa4654ec89a7c207fc4de74194cf"} Nov 03 19:46:04.256157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:04.256120 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:46:04.342120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:04.338987 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" podStartSLOduration=8.338951801 podStartE2EDuration="8.338951801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:04.282452371 +0000 UTC m=+1130.623565857" watchObservedRunningTime="2026-11-03 19:46:04.338951801 +0000 UTC m=+1130.680065287" Nov 03 19:46:05.713713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.713642 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" podStartSLOduration=9.713619897 podStartE2EDuration="9.713619897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:04.417287165 +0000 UTC m=+1130.758400686" watchObservedRunningTime="2026-11-03 19:46:05.713619897 +0000 UTC m=+1132.054733383" Nov 03 19:46:05.717104 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.717077 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh"] Nov 03 19:46:05.729031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.724109 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.730357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.730335 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh"] Nov 03 19:46:05.732247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.732225 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:46:05.733942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.733861 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:46:05.874668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.874632 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.874847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.874719 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.875033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.875005 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqmw\" (UniqueName: \"kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.982321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.976678 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.982321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.976855 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.982321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.977248 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqmw\" (UniqueName: \"kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.982321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.978965 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:05.982705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:05.982613 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:06.001735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:06.001703 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqmw\" (UniqueName: \"kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw\") pod \"maas-api-revoke-sub-a033fd14-9hjrh\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:06.091236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:06.091201 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:06.389571 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:46:06.389524 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d294a6_6078_4e4f_adb0_19910aacdd22.slice/crio-23c3b7e8ec05e5ef18b72ff825f0fe5dfa79e37fd89fac0538ae83e6dc799ccf WatchSource:0}: Error finding container 23c3b7e8ec05e5ef18b72ff825f0fe5dfa79e37fd89fac0538ae83e6dc799ccf: Status 404 returned error can't find the container with id 23c3b7e8ec05e5ef18b72ff825f0fe5dfa79e37fd89fac0538ae83e6dc799ccf Nov 03 19:46:06.392807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:06.392119 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh"] Nov 03 19:46:07.447049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:07.430790 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" event={"ID":"e3d294a6-6078-4e4f-adb0-19910aacdd22","Type":"ContainerStarted","Data":"e24e0add43b38746fc8b81ed0790aadb2ac7e3e6ec963c9d72304f81ed0f2891"} Nov 03 19:46:07.447049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:07.430878 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" event={"ID":"e3d294a6-6078-4e4f-adb0-19910aacdd22","Type":"ContainerStarted","Data":"23c3b7e8ec05e5ef18b72ff825f0fe5dfa79e37fd89fac0538ae83e6dc799ccf"} Nov 03 19:46:07.470988 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:07.468740 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" podStartSLOduration=2.468718774 podStartE2EDuration="2.468718774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:07.457545407 +0000 UTC m=+1133.798658893" watchObservedRunningTime="2026-11-03 19:46:07.468718774 +0000 UTC m=+1133.809832260" Nov 03 19:46:08.461498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:08.460712 5233 generic.go:356] "Generic (PLEG): container finished" podID="e3d294a6-6078-4e4f-adb0-19910aacdd22" containerID="e24e0add43b38746fc8b81ed0790aadb2ac7e3e6ec963c9d72304f81ed0f2891" exitCode=0 Nov 03 19:46:08.461498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:08.461083 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" event={"ID":"e3d294a6-6078-4e4f-adb0-19910aacdd22","Type":"ContainerDied","Data":"e24e0add43b38746fc8b81ed0790aadb2ac7e3e6ec963c9d72304f81ed0f2891"} Nov 03 19:46:10.065405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.065366 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:10.108847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.107627 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca\") pod \"e3d294a6-6078-4e4f-adb0-19910aacdd22\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " Nov 03 19:46:10.108847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.107805 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw\" (UniqueName: \"kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw\") pod \"e3d294a6-6078-4e4f-adb0-19910aacdd22\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " Nov 03 19:46:10.108847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.107849 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca\") pod \"e3d294a6-6078-4e4f-adb0-19910aacdd22\" (UID: \"e3d294a6-6078-4e4f-adb0-19910aacdd22\") " Nov 03 19:46:10.110196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.110163 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca" pod "e3d294a6-6078-4e4f-adb0-19910aacdd22" (UID: "e3d294a6-6078-4e4f-adb0-19910aacdd22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:46:10.159517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.159482 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw" pod "e3d294a6-6078-4e4f-adb0-19910aacdd22" (UID: "e3d294a6-6078-4e4f-adb0-19910aacdd22"). InnerVolumeSpecName "kube-api-access-5lqmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:46:10.159782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.159762 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca" pod "e3d294a6-6078-4e4f-adb0-19910aacdd22" (UID: "e3d294a6-6078-4e4f-adb0-19910aacdd22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:46:10.211447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.211352 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lqmw\" (UniqueName: \"kubernetes.io/projected/e3d294a6-6078-4e4f-adb0-19910aacdd22-kube-api-access-5lqmw\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:10.211447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.211402 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:10.211447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.211421 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d294a6-6078-4e4f-adb0-19910aacdd22-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:10.627831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.627803 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" Nov 03 19:46:10.628139 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.627867 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh" event={"ID":"e3d294a6-6078-4e4f-adb0-19910aacdd22","Type":"ContainerDied","Data":"23c3b7e8ec05e5ef18b72ff825f0fe5dfa79e37fd89fac0538ae83e6dc799ccf"} Nov 03 19:46:10.628228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.628181 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c3b7e8ec05e5ef18b72ff825f0fe5dfa79e37fd89fac0538ae83e6dc799ccf" Nov 03 19:46:10.910583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.910463 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:10.910766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:10.910593 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:13.375885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.375807 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d"] Nov 03 19:46:13.390299 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.376750 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e3d294a6-6078-4e4f-adb0-19910aacdd22" containerName="revoke-keys" Nov 03 19:46:13.390299 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.376767 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d294a6-6078-4e4f-adb0-19910aacdd22" containerName="revoke-keys" Nov 03 19:46:13.390299 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.376908 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d294a6-6078-4e4f-adb0-19910aacdd22" containerName="revoke-keys" Nov 03 19:46:13.412523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.412495 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.419084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.419055 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d"] Nov 03 19:46:13.512175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512139 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbthw\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512272 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512326 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512399 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512479 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512680 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512732 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512819 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512790 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.512940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.512905 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.614570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614531 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614645 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbthw\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614718 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614771 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614820 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614863 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.614939 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615083 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615141 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615204 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615373 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615599 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.615880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615641 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.616062 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.615885 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.618463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.618439 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.618833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.618811 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.628301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.628232 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbthw\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.629418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.629394 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:13.755726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:13.755693 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:14.098824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.098784 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d"] Nov 03 19:46:14.103144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.103109 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:14.103273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.103189 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:14.103273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.103237 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:14.689357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.689319 5233 generic.go:356] "Generic (PLEG): container finished" podID="153c41c1-4143-4168-883d-a6b1af731ce6" containerID="a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd" exitCode=0 Nov 03 19:46:14.689990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.689502 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" event={"ID":"153c41c1-4143-4168-883d-a6b1af731ce6","Type":"ContainerDied","Data":"a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd"} Nov 03 19:46:14.690086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.690066 5233 scope.go:122] "RemoveContainer" containerID="a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd" Nov 03 19:46:14.695251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.693059 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:14.695251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.693152 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:14.695251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.693198 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:14.719509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.719470 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" event={"ID":"8ed3f885-db2f-47e6-abe0-f45a2938e66f","Type":"ContainerStarted","Data":"a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899"} Nov 03 19:46:14.720072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.720044 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" event={"ID":"8ed3f885-db2f-47e6-abe0-f45a2938e66f","Type":"ContainerStarted","Data":"221ed7bc5f16b2dadbb7f2429067db2a1279bdd1ff53d136b2575c4f85fd3e81"} Nov 03 19:46:14.770415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.770387 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:14.778159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:14.778081 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podStartSLOduration=1.778061472 podStartE2EDuration="1.778061472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:14.770008242 +0000 UTC m=+1141.111121729" watchObservedRunningTime="2026-11-03 19:46:14.778061472 +0000 UTC m=+1141.119174959" Nov 03 19:46:15.024155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:15.024071 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:46:15.335589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:15.335495 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:46:15.336691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:15.336672 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:46:15.766245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:15.762650 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:46:15.766245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:15.762702 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 19:46:15.769764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:15.769598 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" event={"ID":"153c41c1-4143-4168-883d-a6b1af731ce6","Type":"ContainerStarted","Data":"8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066"} Nov 03 19:46:16.574997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:16.573345 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:46:16.574997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:16.573464 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 19:46:16.574997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:16.573507 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 19:46:16.756552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:16.756518 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:46:16.756760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:16.756576 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 19:46:17.566995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:17.566336 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 19:46:17.566995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:17.566398 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 19:46:17.757421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:17.756322 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:46:17.757421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:17.756382 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 19:46:18.274789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.274761 5233 scope.go:122] "RemoveContainer" containerID="6293a9fef9b066cc3df992db219c2f27e2fecfee9473c1122a686620724234e4" Nov 03 19:46:18.358207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.357550 5233 scope.go:122] "RemoveContainer" containerID="ac9972d9d58e40bb3bb40a7361c0861b149412d9bff97b10ddad05b2d51fe152" Nov 03 19:46:18.382859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.382837 5233 scope.go:122] "RemoveContainer" containerID="d96e93a97fad56b1389d4fd05b135140369019f140f29a50c405db3d34cd8abc" Nov 03 19:46:18.456804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.456752 5233 scope.go:122] "RemoveContainer" containerID="e655b3de9a2103ad225b74ed38a588474153e899304e0bf33422b71af3092d99" Nov 03 19:46:18.469320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.469188 5233 scope.go:122] "RemoveContainer" containerID="c565dbc76c178bc2cec4b6a008a4e7d16220e9bdf5653af32548841db592b1a1" Nov 03 19:46:18.482270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.482157 5233 scope.go:122] "RemoveContainer" containerID="ac79bb2b1d7eeefb037e4ace1cf4d0e94b0bc30496baa221dbe14a1b6521db87" Nov 03 19:46:18.558090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.553604 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 19:46:18.558090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.553675 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 19:46:18.563679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.563656 5233 scope.go:122] "RemoveContainer" containerID="944401aa2afceac4b254893f943168db4dcf648650e6e3e595493f9612cc30ce" Nov 03 19:46:18.575994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.575957 5233 scope.go:122] "RemoveContainer" containerID="4a9c08a3f672117ddb479c26fecf0e826a21d89697b2f5991ee5fb3fbeab3743" Nov 03 19:46:18.618999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.617667 5233 scope.go:122] "RemoveContainer" containerID="87d6152acecf97b553469ce0088f9c3eb9c12da2e9bbfecaeb0d4a84d4622ef7" Nov 03 19:46:18.758924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.757899 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:46:18.758924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.757999 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 19:46:18.778269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.774905 5233 scope.go:122] "RemoveContainer" containerID="32608cf7e6f022b74f0c47222b32eb4107ec629747df82645a75377fad095e35" Nov 03 19:46:18.802232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.800907 5233 scope.go:122] "RemoveContainer" containerID="a28d3281638899de873c5e6ac48d0f07f403ac662190b847667266be32e85b32" Nov 03 19:46:18.820288 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:18.818560 5233 scope.go:122] "RemoveContainer" containerID="e51f7d1ba3576d72ba2ed51c29fe188ea916951eb222ffa73bb5ecd4edb58d6c" Nov 03 19:46:19.055142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.055110 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm"] Nov 03 19:46:19.063925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.063901 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bacc55f-8dbmm"] Nov 03 19:46:19.560064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.559463 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 19:46:19.560064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.559647 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 19:46:19.757359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.757319 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:46:19.757757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.757380 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 19:46:19.958008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:19.957087 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bb9b9f-e098-402b-b426-31fdac25a95c" path="/var/lib/kubelet/pods/c1bb9b9f-e098-402b-b426-31fdac25a95c/volumes" Nov 03 19:46:20.564093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:20.563483 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 19:46:20.564093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:20.563575 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 19:46:20.757000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:20.756425 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 03 19:46:20.757000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:20.756535 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 03 19:46:21.024994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.024647 5233 generic.go:356] "Generic (PLEG): container finished" podID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerID="0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b" exitCode=0 Nov 03 19:46:21.024994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.024935 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" event={"ID":"1a4b279e-a68c-4dc3-b33f-2da4afa5116d","Type":"ContainerDied","Data":"0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b"} Nov 03 19:46:21.026135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.025674 5233 scope.go:122] "RemoveContainer" containerID="0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b" Nov 03 19:46:21.028382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.028345 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:21.028459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.028442 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:21.028509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.028494 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:21.321395 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:46:21.321208 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:46:21.558007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.555357 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 19:46:21.558007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.555431 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 19:46:21.785001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.773984 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:21.785001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.775560 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:21.785001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:21.780198 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:46:22.038286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.038199 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" event={"ID":"1a4b279e-a68c-4dc3-b33f-2da4afa5116d","Type":"ContainerStarted","Data":"d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5"} Nov 03 19:46:22.144470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.144432 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:46:22.145338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.145309 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:22.145489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.145365 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:22.559825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.559796 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:46:22.561385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.561246 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:46:22.561821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:22.561801 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:46:23.144165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:23.144121 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:23.144686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:23.144201 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:24.152880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:24.152832 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:24.153583 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:24.153552 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:24.869997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:24.869581 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:46:25.155078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:25.144959 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:25.155078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:25.145041 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:26.145993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:26.145541 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:26.145993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:26.145612 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:27.094774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.094715 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:27.114007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.101063 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:27.148113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.147286 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:27.148113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.147347 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:27.274492 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.273684 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:27.383737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.383645 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:27.631120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.620010 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:27.671111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.670998 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:27.935143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:27.935048 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:28.099119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.099068 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:46:28.100380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.100353 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:46:28.102149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.102125 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 19:46:28.114027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.102088 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.114027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.112808 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:46:28.157927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.155347 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:28.157927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.155409 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:28.267998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.240363 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:46:28.692120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.663342 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.692120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.665297 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:46:28.782643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.782589 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:46:28.860653 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.826213 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:46:28.899997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.891362 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:46:28.980055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.902645 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:46:28.980055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.952472 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.980055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.954086 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 19:46:28.980055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.954340 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:46:28.980055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.973218 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:46:28.980055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:28.973347 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:46:29.141075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.140276 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.232301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.232190 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.258464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.236377 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:29.258761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.258726 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:29.525055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.524483 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:46:29.538996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.525457 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:46:29.674388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.674348 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.837915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.837810 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.930003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:29.891378 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:46:30.056989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.055044 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:30.056989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.055391 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:46:30.157092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.156068 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.157692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.157668 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:46:30.175187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.162034 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 19:46:30.252000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.241458 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.442048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.441821 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.503029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.499371 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:46:30.503956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.503767 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:46:30.529418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.529175 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 19:46:30.675252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.675187 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.842867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.842407 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.861113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.849657 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:46:30.861113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.851344 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:46:30.861113 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.855206 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:46:30.878046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.875398 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:46:30.899834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.899065 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:46:30.903052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:30.903026 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:46:31.149331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:31.149254 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:46:31.150523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:31.150503 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:46:34.027489 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.027458 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8"] Nov 03 19:46:34.033480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.033447 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7025055-vwdr8"] Nov 03 19:46:34.381087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.379055 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx"] Nov 03 19:46:34.386079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.383937 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.390939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.390913 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-64454a-openshift-default-dockercfg-zhx28\"" Nov 03 19:46:34.408025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.406120 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx"] Nov 03 19:46:34.449323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.449285 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.449752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.449723 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.450027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.450005 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.450198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.450183 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.450527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.450511 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.450729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.450714 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.450895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.450881 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.451118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.451102 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhzv\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.451329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.451312 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553189 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553231 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553251 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553269 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhzv\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553414 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553524 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553600 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553610 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553694 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553721 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553792 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.553897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.553866 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.554038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.554010 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.554038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.554017 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.555777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.555754 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.556081 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.556063 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.566350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.566328 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.566705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.566683 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhzv\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv\") pod \"e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:34.759143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:34.759112 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:35.093966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:35.092950 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx"] Nov 03 19:46:35.125386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:35.119257 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:35.125386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:35.119354 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:35.125386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:35.119398 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:35.849740 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:35.849702 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ca99ed-8832-494e-8979-a5b3d7e40be9" path="/var/lib/kubelet/pods/e1ca99ed-8832-494e-8979-a5b3d7e40be9/volumes" Nov 03 19:46:36.021997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:36.006934 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" event={"ID":"20b36ad6-bc31-4798-a6b5-e8333c37deb0","Type":"ContainerStarted","Data":"02aee77e5f3a5c9a1365dec5edfac02d56e64b0500f4b9033be5f6b2f8b856ff"} Nov 03 19:46:36.021997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:36.007035 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" event={"ID":"20b36ad6-bc31-4798-a6b5-e8333c37deb0","Type":"ContainerStarted","Data":"a98f9152775eb53a6ad051014776e88fb6a897794f451c7663618c7fbbd399ec"} Nov 03 19:46:36.043363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:36.043304 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podStartSLOduration=2.043289698 podStartE2EDuration="2.043289698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:36.041286535 +0000 UTC m=+1162.382400021" watchObservedRunningTime="2026-11-03 19:46:36.043289698 +0000 UTC m=+1162.384403184" Nov 03 19:46:36.759703 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:36.759671 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:36.761164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:36.761133 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:46:36.761302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:36.761198 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 19:46:37.760272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:37.759918 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:46:37.760272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:37.759987 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 19:46:38.759965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:38.759924 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:46:38.760144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:38.760016 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 19:46:39.760444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:39.760404 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:46:39.760859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:39.760472 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 19:46:40.778135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:40.778096 5233 patch_prober.go:29] interesting pod/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 19:46:40.778736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:40.778698 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 19:46:41.765296 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:41.765263 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:41.765767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:41.765748 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:41.766622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:41.766601 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:46:51.278455 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:46:51.278417 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:46:53.033338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:53.033295 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926"] Nov 03 19:46:53.039632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:53.039603 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c628595-x8926"] Nov 03 19:46:53.839665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:53.839633 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69be721-0fa2-40a2-bea8-a1aa06b99c3d" path="/var/lib/kubelet/pods/a69be721-0fa2-40a2-bea8-a1aa06b99c3d/volumes" Nov 03 19:46:57.027347 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:57.027311 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr"] Nov 03 19:46:57.033211 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:57.033181 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1713bcdb-8lrvr"] Nov 03 19:46:57.839518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:57.839485 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712" path="/var/lib/kubelet/pods/7bca63ef-8a3f-4f44-b21c-bd2d6a3bf712/volumes" Nov 03 19:46:58.520313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.520278 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr"] Nov 03 19:46:58.526016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.525962 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.572316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.572280 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.573755 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.573722 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.573950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.573783 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.573950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.573859 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.573950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.573915 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.574216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.573959 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szr5x\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.574216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.574023 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.574216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.574134 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.574216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.574212 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.584962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.584935 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr"] Nov 03 19:46:58.724575 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.724537 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.724934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.724911 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.725115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.725096 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.725290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.725273 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.725434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.725420 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.725561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.725545 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szr5x\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.740146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.740114 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.740432 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.740414 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.740608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.740596 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.785960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.756280 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.785960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.756500 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.785960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.756742 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.785960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.756966 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.785960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.757218 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.977389 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.977322 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:58.982826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.982732 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szr5x\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:59.003004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:58.999609 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:59.036533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:59.024545 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy\") pod \"e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:59.038301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:59.038273 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:46:59.800506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:59.800446 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr"] Nov 03 19:46:59.812598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:59.812536 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:59.812921 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:59.812890 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:59.813163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:46:59.813136 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:00.704997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:00.704793 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" event={"ID":"2146188c-cb86-422d-9e8a-2e25495d9cd1","Type":"ContainerStarted","Data":"df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5"} Nov 03 19:47:00.704997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:00.704869 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" event={"ID":"2146188c-cb86-422d-9e8a-2e25495d9cd1","Type":"ContainerStarted","Data":"0613a92f99906c9c8169b89efa84b5d304272fc99949808babe77ff6864c6795"} Nov 03 19:47:00.837764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:00.837679 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podStartSLOduration=2.837658296 podStartE2EDuration="2.837658296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:00.748656738 +0000 UTC m=+1187.089770225" watchObservedRunningTime="2026-11-03 19:47:00.837658296 +0000 UTC m=+1187.178771784" Nov 03 19:47:01.047075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.047035 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:47:01.051105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.051073 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:01.051316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.051291 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:01.543997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.543113 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.687242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.687192 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.861199 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.861079 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.862845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.862812 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.893322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.863612 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.948994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:01.942104 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.051655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.051618 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:02.052502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.052466 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:02.077741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.076668 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 03 19:47:02.177182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.177068 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.576000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.567283 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.576000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.568025 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:47:02.778024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.753055 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:47:02.778024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.753646 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:47:02.864049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.863069 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.864546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.864156 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:47:02.916022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.894640 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.916022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.894951 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:47:03.050375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.895213 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:47:03.054001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.050650 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:47:03.076116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.895246 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.077568 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:02.948897 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.104995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.078299 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:47:03.209050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.152825 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.378139 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.251257 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:47:03.458993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.293798 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:47:03.458993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.441936 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:47:03.635263 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.633299 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:03.683812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.683768 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:03.684473 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.657115 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.756992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.753208 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.756992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.753548 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:47:03.756992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.753608 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:47:03.759214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.757698 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" Nov 03 19:47:03.779814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.779783 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:47:03.979460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.974825 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:47:03.979460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.974916 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:47:03.979460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.975006 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:47:03.979460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:03.975069 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:47:04.039126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:04.039089 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:04.039316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:04.039149 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:05.039391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:05.039346 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:05.039864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:05.039407 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:06.039167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.039120 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:06.039383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.039192 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:06.683226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.683187 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4"] Nov 03 19:47:06.752400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.752329 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.760203 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.760101 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4"] Nov 03 19:47:06.765234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.762551 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f292e03c-openshift-default-dockercfg-gbqdz\"" Nov 03 19:47:06.773995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.773381 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.773995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.773537 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.773995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.773595 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.773995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.773799 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6z7\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.773995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.773853 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.774228 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.774293 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.774362 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.774814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.774681 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876419 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876517 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876589 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876669 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876720 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876782 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.876919 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.883376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.881520 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.911119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.890352 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.911119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.890532 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6z7\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.911119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.891211 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.980545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.979233 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.980545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.979819 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:06.980545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:06.980075 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:07.064014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.039299 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:07.074001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.065226 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 19:47:07.074001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.065305 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 19:47:07.074001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.066007 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:07.084240 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.081610 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:07.177993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.161727 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6z7\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7\") pod \"e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:07.380402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.380327 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:07.907582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.881672 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:07.907582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.881747 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:07.907582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.881778 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:07.912286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:07.911791 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4"] Nov 03 19:47:08.045112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:08.045079 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:47:08.045669 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:08.045639 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:47:08.046618 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:08.046597 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:47:08.740386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:08.740346 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" event={"ID":"19e8ce03-af7c-40bd-9980-d40d0184413f","Type":"ContainerStarted","Data":"b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058"} Nov 03 19:47:08.740605 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:08.740426 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" event={"ID":"19e8ce03-af7c-40bd-9980-d40d0184413f","Type":"ContainerStarted","Data":"ad829c52a892b4b2c64092e32c2d6aa90aa2961bd99cfc07275ab1c63b1aeebd"} Nov 03 19:47:08.768812 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:08.768752 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podStartSLOduration=2.768732634 podStartE2EDuration="2.768732634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:08.765409393 +0000 UTC m=+1195.106522883" watchObservedRunningTime="2026-11-03 19:47:08.768732634 +0000 UTC m=+1195.109846120" Nov 03 19:47:09.381744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:09.381715 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:09.388863 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:09.388837 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:09.763433 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:09.763395 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:09.764688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:09.764661 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:10.029155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:10.029080 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn"] Nov 03 19:47:10.034915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:10.034886 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67bfc8fa-tmhvn"] Nov 03 19:47:11.842250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:11.842211 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b34deef-f3d3-4006-a93f-302c977f63aa" path="/var/lib/kubelet/pods/9b34deef-f3d3-4006-a93f-302c977f63aa/volumes" Nov 03 19:47:15.029788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:15.029755 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv"] Nov 03 19:47:15.037277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:15.037249 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1abf6a74-phxkv"] Nov 03 19:47:15.840628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:15.840597 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00fa23d-8cc1-49df-b163-569fb54e26c8" path="/var/lib/kubelet/pods/c00fa23d-8cc1-49df-b163-569fb54e26c8/volumes" Nov 03 19:47:19.029217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.029186 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s"] Nov 03 19:47:19.034997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.034955 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d8691-v2r5s"] Nov 03 19:47:19.273987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.273933 5233 scope.go:122] "RemoveContainer" containerID="bd9f2e9b818403b7dfa0137a374fdc0591cb1b6a6567b71890d21e20326ec53e" Nov 03 19:47:19.291089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.291047 5233 scope.go:122] "RemoveContainer" containerID="9b36d20ec7d03c741518688ed2c5d6914cae0420b8802c4912208dbdbc9e30a1" Nov 03 19:47:19.307048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.307022 5233 scope.go:122] "RemoveContainer" containerID="9db3dee9f90333b144a175a49b20aa73ae219b0c716320d5d05245b993cf6cf2" Nov 03 19:47:19.323332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.323301 5233 scope.go:122] "RemoveContainer" containerID="0626c91e837bbc6626b854b074291382c3d3c5c952cbc75be846604eda4aba6e" Nov 03 19:47:19.351203 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.351169 5233 scope.go:122] "RemoveContainer" containerID="fe8613d97eb0759428be03ed68c0e7972bbc276308a66fa59ef63e6140fad66a" Nov 03 19:47:19.364254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.364222 5233 scope.go:122] "RemoveContainer" containerID="2af3feff03749aeb1e5dc62389634a21287a2f680a94ce7458ad5bf7c554a850" Nov 03 19:47:19.375225 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.375207 5233 scope.go:122] "RemoveContainer" containerID="324ab5f150ab4cb10d8b4840253cae4dfa86687f47767e7c7f5fd8f9067e880f" Nov 03 19:47:19.386144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.386121 5233 scope.go:122] "RemoveContainer" containerID="faa347a103f0d427b226571354719fdcb2ddff044b15ebb72cc240b5543097a6" Nov 03 19:47:19.396269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.396254 5233 scope.go:122] "RemoveContainer" containerID="8df39b38f1cd18dfc88bda8aec9d1ba7b6c992df22ecc9bd11f71c630c339f2c" Nov 03 19:47:19.846331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:19.841588 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a146c62a-01ca-4a08-963c-3d9636225950" path="/var/lib/kubelet/pods/a146c62a-01ca-4a08-963c-3d9636225950/volumes" Nov 03 19:47:21.284368 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:47:21.284336 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:47:22.028202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:22.028168 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj"] Nov 03 19:47:22.034654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:22.034625 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2039ceec-gm2cj"] Nov 03 19:47:23.841212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:23.841141 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb18892-f74e-4a1c-96e6-d39410a5686b" path="/var/lib/kubelet/pods/dfb18892-f74e-4a1c-96e6-d39410a5686b/volumes" Nov 03 19:47:25.034693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:25.031018 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww"] Nov 03 19:47:25.038253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:25.038226 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c2571a-h6qww"] Nov 03 19:47:25.841501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:25.841469 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc78a744-e33c-4fd2-89d1-986f015fa4b6" path="/var/lib/kubelet/pods/cc78a744-e33c-4fd2-89d1-986f015fa4b6/volumes" Nov 03 19:47:34.896327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:34.896291 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j"] Nov 03 19:47:34.966630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:34.963365 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j"] Nov 03 19:47:34.966630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:34.963486 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:34.978246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:34.978118 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:47:34.978246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:34.978218 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:47:35.074289 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.074262 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.074461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.074318 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.074461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.074453 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klxsn\" (UniqueName: \"kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.176086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.176009 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.176234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.176131 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klxsn\" (UniqueName: \"kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.176276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.176266 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.176838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.176816 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.178429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.178407 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.189806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.189777 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klxsn\" (UniqueName: \"kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn\") pod \"maas-api-revoke-sub-9f57f0ea-w6l2j\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.295178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.295150 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:35.619543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:35.619517 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j"] Nov 03 19:47:36.359713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:36.359667 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" event={"ID":"7abf34eb-8aa1-4cb1-8de9-69533130dfda","Type":"ContainerStarted","Data":"84455cd07f63c970d9a1521744eced6ffc6c1ba52a8500821691cc7a668c0811"} Nov 03 19:47:36.360252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:36.359772 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" event={"ID":"7abf34eb-8aa1-4cb1-8de9-69533130dfda","Type":"ContainerStarted","Data":"f6306cf72f7b429dcc8c9a03b91ee067fff6d407da8feeaae2b98821bed0a050"} Nov 03 19:47:36.383899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:36.383847 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" podStartSLOduration=2.38383315 podStartE2EDuration="2.38383315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:36.379084863 +0000 UTC m=+1222.720198345" watchObservedRunningTime="2026-11-03 19:47:36.38383315 +0000 UTC m=+1222.724946634" Nov 03 19:47:37.468052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:37.461907 5233 generic.go:356] "Generic (PLEG): container finished" podID="7abf34eb-8aa1-4cb1-8de9-69533130dfda" containerID="84455cd07f63c970d9a1521744eced6ffc6c1ba52a8500821691cc7a668c0811" exitCode=0 Nov 03 19:47:37.468052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:37.462316 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" event={"ID":"7abf34eb-8aa1-4cb1-8de9-69533130dfda","Type":"ContainerDied","Data":"84455cd07f63c970d9a1521744eced6ffc6c1ba52a8500821691cc7a668c0811"} Nov 03 19:47:38.866148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.866124 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:38.957061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.957025 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn\" (UniqueName: \"kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn\") pod \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " Nov 03 19:47:38.957230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.957213 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca\") pod \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " Nov 03 19:47:38.957486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.957469 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca\") pod \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\" (UID: \"7abf34eb-8aa1-4cb1-8de9-69533130dfda\") " Nov 03 19:47:38.957941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.957877 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca" pod "7abf34eb-8aa1-4cb1-8de9-69533130dfda" (UID: "7abf34eb-8aa1-4cb1-8de9-69533130dfda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:38.958403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.958379 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:38.960286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.960262 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn" pod "7abf34eb-8aa1-4cb1-8de9-69533130dfda" (UID: "7abf34eb-8aa1-4cb1-8de9-69533130dfda"). InnerVolumeSpecName "kube-api-access-klxsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:38.960753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:38.960730 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca" pod "7abf34eb-8aa1-4cb1-8de9-69533130dfda" (UID: "7abf34eb-8aa1-4cb1-8de9-69533130dfda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:39.060135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:39.059988 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abf34eb-8aa1-4cb1-8de9-69533130dfda-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:39.060135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:39.060025 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klxsn\" (UniqueName: \"kubernetes.io/projected/7abf34eb-8aa1-4cb1-8de9-69533130dfda-kube-api-access-klxsn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:39.560311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:39.560287 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" Nov 03 19:47:39.560570 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:39.560323 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j" event={"ID":"7abf34eb-8aa1-4cb1-8de9-69533130dfda","Type":"ContainerDied","Data":"f6306cf72f7b429dcc8c9a03b91ee067fff6d407da8feeaae2b98821bed0a050"} Nov 03 19:47:39.560629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:39.560597 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6306cf72f7b429dcc8c9a03b91ee067fff6d407da8feeaae2b98821bed0a050" Nov 03 19:47:40.084475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:40.084434 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx"] Nov 03 19:47:40.123833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:40.123798 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf76014-8r5wx"] Nov 03 19:47:41.033285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.033248 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8"] Nov 03 19:47:41.040661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.040632 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr"] Nov 03 19:47:41.047767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.047741 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds"] Nov 03 19:47:41.055723 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.055699 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a9c610-469z8"] Nov 03 19:47:41.060654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.060634 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c646bf-zntvr"] Nov 03 19:47:41.065357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.065336 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa7a1fe-vswds"] Nov 03 19:47:41.841075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.841044 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47af1e84-6b91-4dbb-8a09-4880b8dd2825" path="/var/lib/kubelet/pods/47af1e84-6b91-4dbb-8a09-4880b8dd2825/volumes" Nov 03 19:47:41.841674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.841657 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9a689c-bdb4-449a-ad06-39af368f8385" path="/var/lib/kubelet/pods/8d9a689c-bdb4-449a-ad06-39af368f8385/volumes" Nov 03 19:47:41.842243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.842227 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2a2460-49bd-4bd2-a935-d3fd58f497ce" path="/var/lib/kubelet/pods/bb2a2460-49bd-4bd2-a935-d3fd58f497ce/volumes" Nov 03 19:47:41.843024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:41.843006 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe158e4-2340-42e2-a828-f6e0fb869225" path="/var/lib/kubelet/pods/dbe158e4-2340-42e2-a828-f6e0fb869225/volumes" Nov 03 19:47:46.038100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.038064 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8"] Nov 03 19:47:46.045317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.045287 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cfa5ee-7zvc8"] Nov 03 19:47:46.052802 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.052741 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2"] Nov 03 19:47:46.059306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.059281 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2180aeb5-6f9b2"] Nov 03 19:47:46.692519 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.692486 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm"] Nov 03 19:47:46.693714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.693689 5233 cpu_manager.go:414] "containerMap: removing container" podUID="7abf34eb-8aa1-4cb1-8de9-69533130dfda" containerName="revoke-keys" Nov 03 19:47:46.693714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.693716 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abf34eb-8aa1-4cb1-8de9-69533130dfda" containerName="revoke-keys" Nov 03 19:47:46.694016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.693957 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abf34eb-8aa1-4cb1-8de9-69533130dfda" containerName="revoke-keys" Nov 03 19:47:46.748335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.748283 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm"] Nov 03 19:47:46.748513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.748357 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.752600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.752572 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:47:46.754080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.754058 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:47:46.869494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.869457 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.869657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.869505 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfph9\" (UniqueName: \"kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.869756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.869735 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.971628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.971603 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.971860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.971832 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.971959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.971891 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfph9\" (UniqueName: \"kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.972620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.972594 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.974302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.974275 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:46.984634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:46.984605 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfph9\" (UniqueName: \"kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9\") pod \"maas-api-revoke-sub-220955f5-4mxtm\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:47.123247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:47.123203 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:47.911557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:47.911521 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147865af-a377-4379-8f42-c4d1d622ce91" path="/var/lib/kubelet/pods/147865af-a377-4379-8f42-c4d1d622ce91/volumes" Nov 03 19:47:47.912433 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:47.912410 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb30f743-6e82-4fa6-be71-7c79ee23aa1e" path="/var/lib/kubelet/pods/eb30f743-6e82-4fa6-be71-7c79ee23aa1e/volumes" Nov 03 19:47:47.977238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:47.977208 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm"] Nov 03 19:47:47.979380 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:47:47.979347 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef18dce2_81af_4133_972c_183253833e16.slice/crio-a71a82d93c400e66d8fdb0305112231660903435130a07138282dfdeeb440c38 WatchSource:0}: Error finding container a71a82d93c400e66d8fdb0305112231660903435130a07138282dfdeeb440c38: Status 404 returned error can't find the container with id a71a82d93c400e66d8fdb0305112231660903435130a07138282dfdeeb440c38 Nov 03 19:47:48.777435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:48.777399 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" event={"ID":"ef18dce2-81af-4133-972c-183253833e16","Type":"ContainerStarted","Data":"7665e3def4ebb3bc5d509d5de30ba2e883da09a3d7ad417c2ff9b54ec5c34b1d"} Nov 03 19:47:48.777904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:48.777454 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" event={"ID":"ef18dce2-81af-4133-972c-183253833e16","Type":"ContainerStarted","Data":"a71a82d93c400e66d8fdb0305112231660903435130a07138282dfdeeb440c38"} Nov 03 19:47:48.798155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:48.798109 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" podStartSLOduration=2.798094716 podStartE2EDuration="2.798094716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:48.797411374 +0000 UTC m=+1235.138524861" watchObservedRunningTime="2026-11-03 19:47:48.798094716 +0000 UTC m=+1235.139208202" Nov 03 19:47:49.789775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:49.789738 5233 generic.go:356] "Generic (PLEG): container finished" podID="ef18dce2-81af-4133-972c-183253833e16" containerID="7665e3def4ebb3bc5d509d5de30ba2e883da09a3d7ad417c2ff9b54ec5c34b1d" exitCode=0 Nov 03 19:47:49.790284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:49.789826 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" event={"ID":"ef18dce2-81af-4133-972c-183253833e16","Type":"ContainerDied","Data":"7665e3def4ebb3bc5d509d5de30ba2e883da09a3d7ad417c2ff9b54ec5c34b1d"} Nov 03 19:47:51.018301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.018273 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:51.069655 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.069574 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca\") pod \"ef18dce2-81af-4133-972c-183253833e16\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " Nov 03 19:47:51.069818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.069657 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca\") pod \"ef18dce2-81af-4133-972c-183253833e16\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " Nov 03 19:47:51.069818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.069695 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9\" (UniqueName: \"kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9\") pod \"ef18dce2-81af-4133-972c-183253833e16\" (UID: \"ef18dce2-81af-4133-972c-183253833e16\") " Nov 03 19:47:51.070418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.070389 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca" pod "ef18dce2-81af-4133-972c-183253833e16" (UID: "ef18dce2-81af-4133-972c-183253833e16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:51.072572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.072541 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9" pod "ef18dce2-81af-4133-972c-183253833e16" (UID: "ef18dce2-81af-4133-972c-183253833e16"). InnerVolumeSpecName "kube-api-access-xfph9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:51.072948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.072920 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca" pod "ef18dce2-81af-4133-972c-183253833e16" (UID: "ef18dce2-81af-4133-972c-183253833e16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:51.172959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.172926 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef18dce2-81af-4133-972c-183253833e16-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:51.172959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.172960 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef18dce2-81af-4133-972c-183253833e16-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:51.173167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.172989 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfph9\" (UniqueName: \"kubernetes.io/projected/ef18dce2-81af-4133-972c-183253833e16-kube-api-access-xfph9\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:51.359169 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:47:51.359087 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:47:51.818213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.818184 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" Nov 03 19:47:51.818396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.818355 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm" event={"ID":"ef18dce2-81af-4133-972c-183253833e16","Type":"ContainerDied","Data":"a71a82d93c400e66d8fdb0305112231660903435130a07138282dfdeeb440c38"} Nov 03 19:47:51.818452 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:51.818411 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71a82d93c400e66d8fdb0305112231660903435130a07138282dfdeeb440c38" Nov 03 19:47:52.772884 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.772851 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs"] Nov 03 19:47:52.773915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.773891 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ef18dce2-81af-4133-972c-183253833e16" containerName="revoke-keys" Nov 03 19:47:52.773915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.773915 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef18dce2-81af-4133-972c-183253833e16" containerName="revoke-keys" Nov 03 19:47:52.774142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.774116 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef18dce2-81af-4133-972c-183253833e16" containerName="revoke-keys" Nov 03 19:47:52.810067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.809228 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs"] Nov 03 19:47:52.810067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.809387 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:52.816114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.816070 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:47:52.816265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.816124 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:47:52.923717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.923642 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqp9r\" (UniqueName: \"kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:52.923997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.923954 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:52.924305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:52.924279 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.025637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.025554 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.025805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.025649 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.025805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.025789 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqp9r\" (UniqueName: \"kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.026418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.026397 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.028254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.028222 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.038407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.038378 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqp9r\" (UniqueName: \"kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.148788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.148754 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:53.469168 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:47:53.469127 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003a3aaf_a23e_4338_b3ca_e182c13254bf.slice/crio-ec2e67bc4a4c8336f8f285df7b798404fa72e63fe99f10a19d2a3735cd6ada8e WatchSource:0}: Error finding container ec2e67bc4a4c8336f8f285df7b798404fa72e63fe99f10a19d2a3735cd6ada8e: Status 404 returned error can't find the container with id ec2e67bc4a4c8336f8f285df7b798404fa72e63fe99f10a19d2a3735cd6ada8e Nov 03 19:47:53.470948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.470929 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs"] Nov 03 19:47:53.859364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:53.859271 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" event={"ID":"003a3aaf-a23e-4338-b3ca-e182c13254bf","Type":"ContainerStarted","Data":"ec2e67bc4a4c8336f8f285df7b798404fa72e63fe99f10a19d2a3735cd6ada8e"} Nov 03 19:47:54.042545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:54.042499 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4"] Nov 03 19:47:54.058745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:54.058704 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c22b6e-fkzz4"] Nov 03 19:47:54.872816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:54.872742 5233 generic.go:356] "Generic (PLEG): container finished" podID="003a3aaf-a23e-4338-b3ca-e182c13254bf" containerID="20c3c0599c725723d3126b185fbe6cc51939d8f3f57c35e55ff216b97bb8228d" exitCode=0 Nov 03 19:47:54.873209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:54.872826 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" event={"ID":"003a3aaf-a23e-4338-b3ca-e182c13254bf","Type":"ContainerDied","Data":"20c3c0599c725723d3126b185fbe6cc51939d8f3f57c35e55ff216b97bb8228d"} Nov 03 19:47:55.034121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:55.034082 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght"] Nov 03 19:47:55.041542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:55.041505 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2b3c69-glght"] Nov 03 19:47:55.805158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:55.805124 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:55.805357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:55.805180 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:55.841064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:55.841032 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd32916-c92f-4ca6-aac9-69763db76076" path="/var/lib/kubelet/pods/4cd32916-c92f-4ca6-aac9-69763db76076/volumes" Nov 03 19:47:55.841678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:55.841655 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627ceecc-efa8-4556-a6bf-c332db936f47" path="/var/lib/kubelet/pods/627ceecc-efa8-4556-a6bf-c332db936f47/volumes" Nov 03 19:47:56.273993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.262922 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:56.545686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.545401 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca\") pod \"003a3aaf-a23e-4338-b3ca-e182c13254bf\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " Nov 03 19:47:56.545686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.545590 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r\" (UniqueName: \"kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r\") pod \"003a3aaf-a23e-4338-b3ca-e182c13254bf\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " Nov 03 19:47:56.545686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.545665 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca\") pod \"003a3aaf-a23e-4338-b3ca-e182c13254bf\" (UID: \"003a3aaf-a23e-4338-b3ca-e182c13254bf\") " Nov 03 19:47:56.600924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.600863 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r" pod "003a3aaf-a23e-4338-b3ca-e182c13254bf" (UID: "003a3aaf-a23e-4338-b3ca-e182c13254bf"). InnerVolumeSpecName "kube-api-access-lqp9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:56.601342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.601305 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca" pod "003a3aaf-a23e-4338-b3ca-e182c13254bf" (UID: "003a3aaf-a23e-4338-b3ca-e182c13254bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:56.627719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.627671 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca" pod "003a3aaf-a23e-4338-b3ca-e182c13254bf" (UID: "003a3aaf-a23e-4338-b3ca-e182c13254bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:56.648379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.648274 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:56.648379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.648305 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqp9r\" (UniqueName: \"kubernetes.io/projected/003a3aaf-a23e-4338-b3ca-e182c13254bf-kube-api-access-lqp9r\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:56.648379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.648321 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a3aaf-a23e-4338-b3ca-e182c13254bf-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:56.904549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.904465 5233 generic.go:356] "Generic (PLEG): container finished" podID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerID="b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058" exitCode=0 Nov 03 19:47:56.904705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.904538 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" event={"ID":"19e8ce03-af7c-40bd-9980-d40d0184413f","Type":"ContainerDied","Data":"b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058"} Nov 03 19:47:56.905090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.905070 5233 scope.go:122] "RemoveContainer" containerID="b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058" Nov 03 19:47:56.906571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.906548 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" event={"ID":"003a3aaf-a23e-4338-b3ca-e182c13254bf","Type":"ContainerDied","Data":"ec2e67bc4a4c8336f8f285df7b798404fa72e63fe99f10a19d2a3735cd6ada8e"} Nov 03 19:47:56.906677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.906595 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2e67bc4a4c8336f8f285df7b798404fa72e63fe99f10a19d2a3735cd6ada8e" Nov 03 19:47:56.906677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.906657 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs" Nov 03 19:47:56.907282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.907246 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:56.907408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.907338 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:56.907408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:56.907387 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:47:57.938995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:57.937495 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" event={"ID":"19e8ce03-af7c-40bd-9980-d40d0184413f","Type":"ContainerStarted","Data":"91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48"} Nov 03 19:47:58.046253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:58.046220 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm"] Nov 03 19:47:58.057663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:58.056751 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39e28e5c-q6rqm"] Nov 03 19:47:58.381484 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:58.381443 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:47:58.382612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:58.382586 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:47:58.382750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:58.382635 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:47:59.034625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.034588 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7"] Nov 03 19:47:59.040900 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.040868 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n"] Nov 03 19:47:59.046167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.046132 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1594089-44gs7"] Nov 03 19:47:59.051249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.051228 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d1ab60-bxq2n"] Nov 03 19:47:59.382133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.382039 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:47:59.382133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.382106 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:47:59.881650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.881619 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5262db5c-5e88-47a6-8d97-5dff20a54984" path="/var/lib/kubelet/pods/5262db5c-5e88-47a6-8d97-5dff20a54984/volumes" Nov 03 19:47:59.902888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.902816 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea7d8dd-cb5b-44ca-8128-2820db2e5453" path="/var/lib/kubelet/pods/5ea7d8dd-cb5b-44ca-8128-2820db2e5453/volumes" Nov 03 19:47:59.906117 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:47:59.906094 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625e5f15-958c-4d23-aae7-43b58adbddfb" path="/var/lib/kubelet/pods/625e5f15-958c-4d23-aae7-43b58adbddfb/volumes" Nov 03 19:48:00.381827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:00.381791 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:48:00.382245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:00.381864 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:48:01.067932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.067894 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx"] Nov 03 19:48:01.068842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.068818 5233 cpu_manager.go:414] "containerMap: removing container" podUID="003a3aaf-a23e-4338-b3ca-e182c13254bf" containerName="revoke-keys" Nov 03 19:48:01.068842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.068843 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003a3aaf-a23e-4338-b3ca-e182c13254bf" containerName="revoke-keys" Nov 03 19:48:01.069105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.069088 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="003a3aaf-a23e-4338-b3ca-e182c13254bf" containerName="revoke-keys" Nov 03 19:48:01.072153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.072131 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.085204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.085172 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.085671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.085646 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kjl\" (UniqueName: \"kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl\") pod \"payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.152991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.152947 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-29ab98\"" Nov 03 19:48:01.165499 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.165452 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-29ab98-dockercfg-vh2jv\"" Nov 03 19:48:01.230994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.195187 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kjl\" (UniqueName: \"kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl\") pod \"payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.254670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.254622 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.256340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.256316 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.426505 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.425404 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:48:01.426505 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.425477 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:48:01.523422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.459766 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kjl\" (UniqueName: \"kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl\") pod \"payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.523422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.491346 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx"] Nov 03 19:48:01.591017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.568470 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:01.978340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:01.974520 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx"] Nov 03 19:48:02.074236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:02.074200 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" event={"ID":"8aeac680-38eb-4ee0-8173-fe496a7a0b11","Type":"ContainerStarted","Data":"034c9cdee03ec5d9fc8839b255c97b4bda1fe29c8a9cc30a3d4b9209f3a03acc"} Nov 03 19:48:02.382513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:02.382419 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:48:02.382513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:02.382480 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:48:02.952339 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:02.951059 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9"] Nov 03 19:48:02.967509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:02.957190 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd"] Nov 03 19:48:03.141406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:03.138964 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" event={"ID":"8aeac680-38eb-4ee0-8173-fe496a7a0b11","Type":"ContainerStarted","Data":"5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29"} Nov 03 19:48:03.141406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:03.139347 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:03.166287 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:03.166233 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" podStartSLOduration=2.166217135 podStartE2EDuration="2.166217135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:03.158492083 +0000 UTC m=+1249.499605581" watchObservedRunningTime="2026-11-03 19:48:03.166217135 +0000 UTC m=+1249.507330619" Nov 03 19:48:03.403003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:03.390351 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:48:03.403003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:03.390412 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:48:04.386375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:04.386335 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:48:04.386857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:04.386834 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:48:04.387485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:04.387459 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:48:05.179677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.179637 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7"] Nov 03 19:48:05.184801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.184767 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.189333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.189292 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:48:05.190737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.190715 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:48:05.210694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.210668 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7"] Nov 03 19:48:05.319905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.319873 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.320090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.319917 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qc4r\" (UniqueName: \"kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.320148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.320131 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.446250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.444858 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.446250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.445242 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.446250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.445289 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qc4r\" (UniqueName: \"kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.469087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.468128 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.484246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.484177 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qc4r\" (UniqueName: \"kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.495000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.488077 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb4e655-d5sd7\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.509183 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.509156 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:05.847800 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:48:05.847769 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ae4820_54c7_4ec5_b9f9_fbc50ac9558b.slice/crio-421d09810c5dbaf9a0be806ebe6f2db736116f9eb136789e88e00954b4e9a22b WatchSource:0}: Error finding container 421d09810c5dbaf9a0be806ebe6f2db736116f9eb136789e88e00954b4e9a22b: Status 404 returned error can't find the container with id 421d09810c5dbaf9a0be806ebe6f2db736116f9eb136789e88e00954b4e9a22b Nov 03 19:48:05.848958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:05.848933 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7"] Nov 03 19:48:06.176613 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:06.176577 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" event={"ID":"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b","Type":"ContainerStarted","Data":"421d09810c5dbaf9a0be806ebe6f2db736116f9eb136789e88e00954b4e9a22b"} Nov 03 19:48:07.189581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:07.189491 5233 generic.go:356] "Generic (PLEG): container finished" podID="d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" containerID="4c171cd1f51dbd7f9abf636be8ba48c39cf7678992d4c8c6d62d6ae89985c5ac" exitCode=0 Nov 03 19:48:07.190101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:07.189580 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" event={"ID":"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b","Type":"ContainerDied","Data":"4c171cd1f51dbd7f9abf636be8ba48c39cf7678992d4c8c6d62d6ae89985c5ac"} Nov 03 19:48:07.952226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:07.952158 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" podUID="4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" containerName="payload-processing" containerID="cri-o://1d5cbbd0978d42a3a31e0dee001163a839b4fa4654ec89a7c207fc4de74194cf" gracePeriod=25 Nov 03 19:48:07.957712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:07.957650 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" podUID="06c01ec7-1fa5-43e1-8d49-facc35c28250" containerName="payload-pre-processing" containerID="cri-o://95768a387f9a05aee8bc167b67c1b9598b101aef332960e3bf5e545db9c569ef" gracePeriod=25 Nov 03 19:48:08.271932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.271896 5233 generic.go:356] "Generic (PLEG): container finished" podID="06c01ec7-1fa5-43e1-8d49-facc35c28250" containerID="95768a387f9a05aee8bc167b67c1b9598b101aef332960e3bf5e545db9c569ef" exitCode=0 Nov 03 19:48:08.272439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.272132 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" event={"ID":"06c01ec7-1fa5-43e1-8d49-facc35c28250","Type":"ContainerDied","Data":"95768a387f9a05aee8bc167b67c1b9598b101aef332960e3bf5e545db9c569ef"} Nov 03 19:48:08.274693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.274666 5233 generic.go:356] "Generic (PLEG): container finished" podID="4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" containerID="1d5cbbd0978d42a3a31e0dee001163a839b4fa4654ec89a7c207fc4de74194cf" exitCode=0 Nov 03 19:48:08.275033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.274955 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" event={"ID":"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8","Type":"ContainerDied","Data":"1d5cbbd0978d42a3a31e0dee001163a839b4fa4654ec89a7c207fc4de74194cf"} Nov 03 19:48:08.424089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.424058 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:48:08.461551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.461524 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:48:08.536075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.526739 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:08.618344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.618293 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp\" (UniqueName: \"kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp\") pod \"06c01ec7-1fa5-43e1-8d49-facc35c28250\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " Nov 03 19:48:08.618551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.618389 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume\") pod \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " Nov 03 19:48:08.618551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.618499 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8\" (UniqueName: \"kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8\") pod \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\" (UID: \"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8\") " Nov 03 19:48:08.618677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.618578 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume\") pod \"06c01ec7-1fa5-43e1-8d49-facc35c28250\" (UID: \"06c01ec7-1fa5-43e1-8d49-facc35c28250\") " Nov 03 19:48:08.618939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.618833 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume" pod "4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" (UID: "4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:48:08.619102 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.619060 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume" pod "06c01ec7-1fa5-43e1-8d49-facc35c28250" (UID: "06c01ec7-1fa5-43e1-8d49-facc35c28250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:48:08.620122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.619507 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:08.620122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.619569 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c01ec7-1fa5-43e1-8d49-facc35c28250-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:08.621627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.621598 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp" pod "06c01ec7-1fa5-43e1-8d49-facc35c28250" (UID: "06c01ec7-1fa5-43e1-8d49-facc35c28250"). InnerVolumeSpecName "kube-api-access-cxhvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:48:08.621983 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.621953 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8" pod "4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" (UID: "4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8"). InnerVolumeSpecName "kube-api-access-brml8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:48:08.721167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.721132 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca\") pod \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " Nov 03 19:48:08.721370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.721292 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r\" (UniqueName: \"kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r\") pod \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " Nov 03 19:48:08.744029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.741716 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca" pod "d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" (UID: "d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:48:08.744029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.741777 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca\") pod \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\" (UID: \"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b\") " Nov 03 19:48:08.744029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.742986 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca" pod "d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" (UID: "d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:48:08.746250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.746216 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r" pod "d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" (UID: "d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b"). InnerVolumeSpecName "kube-api-access-9qc4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:48:08.767995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.767558 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:08.767995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.767595 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxhvp\" (UniqueName: \"kubernetes.io/projected/06c01ec7-1fa5-43e1-8d49-facc35c28250-kube-api-access-cxhvp\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:08.767995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.767611 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:08.767995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.767628 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brml8\" (UniqueName: \"kubernetes.io/projected/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8-kube-api-access-brml8\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:08.767995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:08.767642 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qc4r\" (UniqueName: \"kubernetes.io/projected/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b-kube-api-access-9qc4r\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:09.289677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.289607 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" Nov 03 19:48:09.290108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.289752 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9" event={"ID":"4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8","Type":"ContainerDied","Data":"83170f97f65ca0a24d7dbfef044b464c78c7210ef56d070e45688e3210a33af4"} Nov 03 19:48:09.290108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.289816 5233 scope.go:122] "RemoveContainer" containerID="1d5cbbd0978d42a3a31e0dee001163a839b4fa4654ec89a7c207fc4de74194cf" Nov 03 19:48:09.296629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.296371 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" Nov 03 19:48:09.296629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.296539 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7" event={"ID":"d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b","Type":"ContainerDied","Data":"421d09810c5dbaf9a0be806ebe6f2db736116f9eb136789e88e00954b4e9a22b"} Nov 03 19:48:09.296629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.296586 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421d09810c5dbaf9a0be806ebe6f2db736116f9eb136789e88e00954b4e9a22b" Nov 03 19:48:09.302721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.302701 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" Nov 03 19:48:09.302827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.302732 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd" event={"ID":"06c01ec7-1fa5-43e1-8d49-facc35c28250","Type":"ContainerDied","Data":"de6853ac274f559efad75341bab270efe1613069d81246329326ae5e022d1af1"} Nov 03 19:48:09.307922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.305687 5233 scope.go:122] "RemoveContainer" containerID="95768a387f9a05aee8bc167b67c1b9598b101aef332960e3bf5e545db9c569ef" Nov 03 19:48:09.325623 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.325598 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9"] Nov 03 19:48:09.350826 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.341206 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-mcgp9"] Nov 03 19:48:09.443452 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.442438 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd"] Nov 03 19:48:09.451227 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.451202 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-49cpd"] Nov 03 19:48:09.867997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.864908 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c01ec7-1fa5-43e1-8d49-facc35c28250" path="/var/lib/kubelet/pods/06c01ec7-1fa5-43e1-8d49-facc35c28250/volumes" Nov 03 19:48:09.867997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:09.865651 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" path="/var/lib/kubelet/pods/4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8/volumes" Nov 03 19:48:10.039061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:10.038282 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx"] Nov 03 19:48:10.043871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:10.043794 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8621181-qwhlx"] Nov 03 19:48:11.892990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:11.892160 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdad68d5-09d7-43ee-9120-cf5d27dbb80e" path="/var/lib/kubelet/pods/cdad68d5-09d7-43ee-9120-cf5d27dbb80e/volumes" Nov 03 19:48:12.030851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:12.030812 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d"] Nov 03 19:48:12.036324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:12.036295 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84c6e31-5xr4d"] Nov 03 19:48:13.841554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:13.841525 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb34026-7268-4264-8e3c-420b52c65890" path="/var/lib/kubelet/pods/4cb34026-7268-4264-8e3c-420b52c65890/volumes" Nov 03 19:48:14.030176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:14.030140 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52"] Nov 03 19:48:14.037461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:14.037432 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a3d46c-mcf52"] Nov 03 19:48:14.153509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:14.153436 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:48:15.849545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:15.849502 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218c7627-7a47-4f87-b520-a04f66eaa43a" path="/var/lib/kubelet/pods/218c7627-7a47-4f87-b520-a04f66eaa43a/volumes" Nov 03 19:48:16.026793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:16.026753 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7"] Nov 03 19:48:16.032111 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:16.032080 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1150360e-5tdw7"] Nov 03 19:48:18.038003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.013687 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b842fb1-0d91-4240-b380-f0e085faf9e5" path="/var/lib/kubelet/pods/8b842fb1-0d91-4240-b380-f0e085faf9e5/volumes" Nov 03 19:48:18.686994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.686078 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:18.787995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.785062 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:18.787995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.786205 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:18.787995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.786702 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:18.885103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.885056 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:18.890604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.890570 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:48:18.936123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:18.936079 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:19.498995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.498166 5233 scope.go:122] "RemoveContainer" containerID="64c7f008ddfa7aeddb5a161ce7eb1ade478cceaeb4a3c71fe0209143b25d10ca" Nov 03 19:48:19.606276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.603139 5233 scope.go:122] "RemoveContainer" containerID="5d773f312dc5b0d91ed88c492c9478d93aa85698fbd5a5efbc9043c46551a750" Nov 03 19:48:19.725638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.725611 5233 scope.go:122] "RemoveContainer" containerID="fd86cabf65cefbcfe7f4e35d947e8a53844269da6d293a499bf836faba0314c7" Nov 03 19:48:19.757700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.757672 5233 scope.go:122] "RemoveContainer" containerID="c603d79f6b12fa8e0e8ef089c2323a43463ddefd3df42274c8228ebd6721b646" Nov 03 19:48:19.778877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.778054 5233 scope.go:122] "RemoveContainer" containerID="907d2a12b9b2d7a046ff8d9739ec128ec37132a7e1b61102b92ed42139ae81c2" Nov 03 19:48:19.797620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.797315 5233 scope.go:122] "RemoveContainer" containerID="581e7a886e4cb112d2b6a04ca5efc29c147d730bc428517d1c6f68ebefe5b105" Nov 03 19:48:19.812324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:19.812211 5233 scope.go:122] "RemoveContainer" containerID="8aa8840b88260d5096d35668e3a2a1f6e4fd21b391789c656ea8e72ad13237bc" Nov 03 19:48:21.279603 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:48:21.279572 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:48:22.053391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:22.053354 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs"] Nov 03 19:48:22.069951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:22.061925 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-7frqs"] Nov 03 19:48:23.032391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:23.032351 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz"] Nov 03 19:48:23.043867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:23.043834 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1afcc4c7-2nrhz"] Nov 03 19:48:23.841016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:23.840962 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003a3aaf-a23e-4338-b3ca-e182c13254bf" path="/var/lib/kubelet/pods/003a3aaf-a23e-4338-b3ca-e182c13254bf/volumes" Nov 03 19:48:23.841495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:23.841481 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386e90a1-4f59-4eaf-9913-035edce32dad" path="/var/lib/kubelet/pods/386e90a1-4f59-4eaf-9913-035edce32dad/volumes" Nov 03 19:48:24.027582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:24.027547 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb"] Nov 03 19:48:24.032457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:24.032427 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4a982f-rrfsb"] Nov 03 19:48:25.841770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:25.841732 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b2eda3-01bb-4529-a93c-11475c5a11af" path="/var/lib/kubelet/pods/70b2eda3-01bb-4529-a93c-11475c5a11af/volumes" Nov 03 19:48:28.030699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:28.030669 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr"] Nov 03 19:48:28.037080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:28.037055 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7290e24c-9jrbr"] Nov 03 19:48:29.840524 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:29.840483 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2f44f8-7025-407e-a9c3-a83d7f2af1c2" path="/var/lib/kubelet/pods/aa2f44f8-7025-407e-a9c3-a83d7f2af1c2/volumes" Nov 03 19:48:30.041300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.041259 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw"] Nov 03 19:48:30.047631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.047594 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744125b-f9wcw"] Nov 03 19:48:30.461374 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.461344 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf"] Nov 03 19:48:30.462226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462203 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" containerName="revoke-keys" Nov 03 19:48:30.462226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462222 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" containerName="revoke-keys" Nov 03 19:48:30.462226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462233 5233 cpu_manager.go:414] "containerMap: removing container" podUID="06c01ec7-1fa5-43e1-8d49-facc35c28250" containerName="payload-pre-processing" Nov 03 19:48:30.462456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462239 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c01ec7-1fa5-43e1-8d49-facc35c28250" containerName="payload-pre-processing" Nov 03 19:48:30.462456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462258 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" containerName="payload-processing" Nov 03 19:48:30.462456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462263 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" containerName="payload-processing" Nov 03 19:48:30.462456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462434 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" containerName="revoke-keys" Nov 03 19:48:30.462456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462453 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c01ec7-1fa5-43e1-8d49-facc35c28250" containerName="payload-pre-processing" Nov 03 19:48:30.462739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.462466 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccdc5f7-b009-4c89-8c7d-734b1bd9d9a8" containerName="payload-processing" Nov 03 19:48:30.466469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.466450 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.475069 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.475042 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eae28ff0\"" Nov 03 19:48:30.477663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.477637 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eae28ff0-dockercfg-ds2dl\"" Nov 03 19:48:30.483512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.483488 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h"] Nov 03 19:48:30.520215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.518128 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf"] Nov 03 19:48:30.520215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.518262 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:30.535317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.535281 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h"] Nov 03 19:48:30.541696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.541667 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrrz\" (UniqueName: \"kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.541869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.541814 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5wh\" (UniqueName: \"kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:30.541967 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.541888 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:30.542087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.542013 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.647993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.645230 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.647993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.645448 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrrz\" (UniqueName: \"kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.647993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.645598 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5wh\" (UniqueName: \"kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:30.647993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.645675 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:30.647993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.646662 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:30.647993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.646851 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.912998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.895469 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrrz\" (UniqueName: \"kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz\") pod \"payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:30.987692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:30.987658 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5wh\" (UniqueName: \"kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh\") pod \"payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:31.136886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:31.136858 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:31.145135 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:31.145103 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:31.476127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:31.476079 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h"] Nov 03 19:48:31.491787 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:31.491759 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf"] Nov 03 19:48:32.011641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:32.011597 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfd816b-4816-40bc-955f-bdb17922f3fa" path="/var/lib/kubelet/pods/5dfd816b-4816-40bc-955f-bdb17922f3fa/volumes" Nov 03 19:48:32.079332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:32.079297 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" event={"ID":"85a99088-473f-40f5-b963-e3577c99fea4","Type":"ContainerStarted","Data":"f5bf8b9eaf6eefe3dc65360aea58e451e01a43f4acae4e5bcf78028dac63529f"} Nov 03 19:48:32.080992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:32.080940 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" event={"ID":"4779d85c-759c-464f-b860-c28044707aa6","Type":"ContainerStarted","Data":"c5c6394848a665a5d93802a4c3fe05d1eea549ff2ba78075108753b71376030b"} Nov 03 19:48:33.110186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:33.110144 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" event={"ID":"85a99088-473f-40f5-b963-e3577c99fea4","Type":"ContainerStarted","Data":"e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a"} Nov 03 19:48:33.110691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:33.110606 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:33.114029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:33.113303 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" event={"ID":"4779d85c-759c-464f-b860-c28044707aa6","Type":"ContainerStarted","Data":"2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0"} Nov 03 19:48:33.114029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:33.113462 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:33.135544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:33.135489 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" podStartSLOduration=3.135472492 podStartE2EDuration="3.135472492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:33.13220202 +0000 UTC m=+1279.473315505" watchObservedRunningTime="2026-11-03 19:48:33.135472492 +0000 UTC m=+1279.476585977" Nov 03 19:48:33.159455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:33.159405 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" podStartSLOduration=3.159389495 podStartE2EDuration="3.159389495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:33.154330322 +0000 UTC m=+1279.495443808" watchObservedRunningTime="2026-11-03 19:48:33.159389495 +0000 UTC m=+1279.500502987" Nov 03 19:48:43.056164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:43.055497 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n"] Nov 03 19:48:43.060582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:43.060553 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df54c40-4sr5n"] Nov 03 19:48:43.841083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:43.841052 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3829410-2108-4749-8b62-d32ee82b7e99" path="/var/lib/kubelet/pods/e3829410-2108-4749-8b62-d32ee82b7e99/volumes" Nov 03 19:48:44.125532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:44.125448 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:48:44.125935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:44.125832 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:48:46.029201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:46.029157 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl"] Nov 03 19:48:46.034907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:46.034879 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts"] Nov 03 19:48:46.039202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:46.039184 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67f295a-hxjkl"] Nov 03 19:48:46.043283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:46.043260 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d261ab-zwmts"] Nov 03 19:48:47.955996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:47.927070 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb50808-1204-404f-83c6-3125bddee698" path="/var/lib/kubelet/pods/0fb50808-1204-404f-83c6-3125bddee698/volumes" Nov 03 19:48:47.957521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:47.957492 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98a9b0a-1de9-4a96-95b7-82a72f313b19" path="/var/lib/kubelet/pods/f98a9b0a-1de9-4a96-95b7-82a72f313b19/volumes" Nov 03 19:48:49.029547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:49.029507 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4"] Nov 03 19:48:49.037225 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:49.037194 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d736b56-dfks4"] Nov 03 19:48:49.843797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:49.843765 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbced5b-4b7e-496d-9461-dd99fe6aadd5" path="/var/lib/kubelet/pods/6bbced5b-4b7e-496d-9461-dd99fe6aadd5/volumes" Nov 03 19:48:51.279376 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:48:51.279340 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:48:56.436571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.436489 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz"] Nov 03 19:48:56.444531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.444507 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.458056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.458003 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz"] Nov 03 19:48:56.459439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.459077 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:48:56.460110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.460076 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:48:56.492666 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.492634 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.493108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.493077 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.493271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.493250 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdbz\" (UniqueName: \"kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.597294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.597250 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.597582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.597517 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.597582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.597568 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdbz\" (UniqueName: \"kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.599165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.599123 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.601185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.601156 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.638226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.638190 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdbz\" (UniqueName: \"kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz\") pod \"maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:56.800329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:56.800289 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:48:57.436116 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:48:57.436083 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3f8254_5980_4e2f_bf89_bb8ae84d3bc9.slice/crio-e92b2d8aa415b15752846c7a2ff822f9f5009af42e033901dd88af96491c2b37 WatchSource:0}: Error finding container e92b2d8aa415b15752846c7a2ff822f9f5009af42e033901dd88af96491c2b37: Status 404 returned error can't find the container with id e92b2d8aa415b15752846c7a2ff822f9f5009af42e033901dd88af96491c2b37 Nov 03 19:48:57.436116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:57.436092 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz"] Nov 03 19:48:57.760640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:57.760590 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" event={"ID":"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9","Type":"ContainerStarted","Data":"e92b2d8aa415b15752846c7a2ff822f9f5009af42e033901dd88af96491c2b37"} Nov 03 19:48:58.859025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:58.858921 5233 generic.go:356] "Generic (PLEG): container finished" podID="6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" containerID="3f2afe26eb9189ee2f3662a502d3e5b756f6ee0ad2c5c415af6b2d28446ea11c" exitCode=0 Nov 03 19:48:58.859644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:48:58.859614 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" event={"ID":"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9","Type":"ContainerDied","Data":"3f2afe26eb9189ee2f3662a502d3e5b756f6ee0ad2c5c415af6b2d28446ea11c"} Nov 03 19:49:00.102994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.101473 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:49:00.139044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.138710 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz\" (UniqueName: \"kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz\") pod \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " Nov 03 19:49:00.139215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.139071 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca\") pod \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " Nov 03 19:49:00.139215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.139196 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca\") pod \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\" (UID: \"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9\") " Nov 03 19:49:00.141374 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.140462 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca" pod "6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" (UID: "6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:00.144995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.142733 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz" pod "6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" (UID: "6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9"). InnerVolumeSpecName "kube-api-access-lqdbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:00.144995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.143018 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca" pod "6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" (UID: "6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:00.242349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.242315 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqdbz\" (UniqueName: \"kubernetes.io/projected/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-kube-api-access-lqdbz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:00.242506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.242353 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:00.242506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.242371 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:00.886535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.886502 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" event={"ID":"6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9","Type":"ContainerDied","Data":"e92b2d8aa415b15752846c7a2ff822f9f5009af42e033901dd88af96491c2b37"} Nov 03 19:49:00.886815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.886781 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e92b2d8aa415b15752846c7a2ff822f9f5009af42e033901dd88af96491c2b37" Nov 03 19:49:00.886815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:00.886514 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz" Nov 03 19:49:01.953764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:01.953731 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc"] Nov 03 19:49:01.954537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:01.954519 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" containerName="revoke-keys" Nov 03 19:49:01.954537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:01.954534 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" containerName="revoke-keys" Nov 03 19:49:01.954702 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:01.954661 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" containerName="revoke-keys" Nov 03 19:49:01.958547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:01.958521 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:01.966195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:01.966173 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc"] Nov 03 19:49:02.050012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.049483 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-10e551\"" Nov 03 19:49:02.138048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.096649 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcrzn\" (UniqueName: \"kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn\") pod \"payload-processing-e2e-api-a-10e551-646757c557-tttqc\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.138048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.097031 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10e551-646757c557-tttqc\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.138328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.030406 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-10e551-dockercfg-7gndr\"" Nov 03 19:49:02.256058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.250200 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcrzn\" (UniqueName: \"kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn\") pod \"payload-processing-e2e-api-a-10e551-646757c557-tttqc\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.275490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.271557 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10e551-646757c557-tttqc\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.301016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.298165 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10e551-646757c557-tttqc\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.379025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.346470 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcrzn\" (UniqueName: \"kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn\") pod \"payload-processing-e2e-api-a-10e551-646757c557-tttqc\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.591699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.591272 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:02.935513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:02.935482 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc"] Nov 03 19:49:02.936252 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:49:02.936228 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a61bca2_e418_4af1_a018_3b377295f94e.slice/crio-efd8df8bd59bbc432fc467b40f5640316810868162ea304b2c99842cd6621dec WatchSource:0}: Error finding container efd8df8bd59bbc432fc467b40f5640316810868162ea304b2c99842cd6621dec: Status 404 returned error can't find the container with id efd8df8bd59bbc432fc467b40f5640316810868162ea304b2c99842cd6621dec Nov 03 19:49:03.975912 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:03.975869 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" event={"ID":"7a61bca2-e418-4af1-a018-3b377295f94e","Type":"ContainerStarted","Data":"1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2"} Nov 03 19:49:03.976529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:03.975939 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" event={"ID":"7a61bca2-e418-4af1-a018-3b377295f94e","Type":"ContainerStarted","Data":"efd8df8bd59bbc432fc467b40f5640316810868162ea304b2c99842cd6621dec"} Nov 03 19:49:03.976712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:03.976690 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:04.012190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:04.012123 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" podStartSLOduration=3.012102339 podStartE2EDuration="3.012102339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:04.003937618 +0000 UTC m=+1310.345051104" watchObservedRunningTime="2026-11-03 19:49:04.012102339 +0000 UTC m=+1310.353215825" Nov 03 19:49:06.592995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:06.588233 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h"] Nov 03 19:49:06.617272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:06.608108 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf"] Nov 03 19:49:08.675987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.675944 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7"] Nov 03 19:49:08.681119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.681100 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.691334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.690918 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:49:08.691334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.691004 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:49:08.705020 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.700936 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7"] Nov 03 19:49:08.804660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.804619 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.804865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.804736 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5frzt\" (UniqueName: \"kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.804984 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.804950 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.906889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.906850 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.906889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.906893 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5frzt\" (UniqueName: \"kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.907160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.907052 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.907705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.907680 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.909521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.909493 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:08.919618 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:08.919591 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5frzt\" (UniqueName: \"kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt\") pod \"maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:09.003078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:09.003043 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:09.387950 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:49:09.387915 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3abc8b_2acf_41bd_9b11_3d3be8cf4fe7.slice/crio-1523fb0301bc274b588e7dcdee78452a840fd7a43f4d6a08ad3cdbfbd8b0b449 WatchSource:0}: Error finding container 1523fb0301bc274b588e7dcdee78452a840fd7a43f4d6a08ad3cdbfbd8b0b449: Status 404 returned error can't find the container with id 1523fb0301bc274b588e7dcdee78452a840fd7a43f4d6a08ad3cdbfbd8b0b449 Nov 03 19:49:09.388206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:09.388184 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7"] Nov 03 19:49:10.150951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:10.150915 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" event={"ID":"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7","Type":"ContainerStarted","Data":"9219317d05eb4b45affdc286c9af98ca1c5aad9e2ccf5abd1767ee6ffb250a1f"} Nov 03 19:49:10.151356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:10.150981 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" event={"ID":"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7","Type":"ContainerStarted","Data":"1523fb0301bc274b588e7dcdee78452a840fd7a43f4d6a08ad3cdbfbd8b0b449"} Nov 03 19:49:10.172986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:10.172935 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" podStartSLOduration=2.172923198 podStartE2EDuration="2.172923198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:10.168670899 +0000 UTC m=+1316.509784384" watchObservedRunningTime="2026-11-03 19:49:10.172923198 +0000 UTC m=+1316.514036682" Nov 03 19:49:11.163679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:11.163640 5233 generic.go:356] "Generic (PLEG): container finished" podID="ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" containerID="9219317d05eb4b45affdc286c9af98ca1c5aad9e2ccf5abd1767ee6ffb250a1f" exitCode=0 Nov 03 19:49:11.164206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:11.163729 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" event={"ID":"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7","Type":"ContainerDied","Data":"9219317d05eb4b45affdc286c9af98ca1c5aad9e2ccf5abd1767ee6ffb250a1f"} Nov 03 19:49:11.589639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:11.589566 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" podUID="85a99088-473f-40f5-b963-e3577c99fea4" containerName="payload-processing" containerID="cri-o://e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a" gracePeriod=25 Nov 03 19:49:11.610490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:11.609983 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" podUID="4779d85c-759c-464f-b860-c28044707aa6" containerName="payload-pre-processing" containerID="cri-o://2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0" gracePeriod=25 Nov 03 19:49:11.966811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:11.966776 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:49:12.031386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.031349 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume\") pod \"85a99088-473f-40f5-b963-e3577c99fea4\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " Nov 03 19:49:12.031576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.031506 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh\" (UniqueName: \"kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh\") pod \"85a99088-473f-40f5-b963-e3577c99fea4\" (UID: \"85a99088-473f-40f5-b963-e3577c99fea4\") " Nov 03 19:49:12.033123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.033093 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume" pod "85a99088-473f-40f5-b963-e3577c99fea4" (UID: "85a99088-473f-40f5-b963-e3577c99fea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:12.035610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.035518 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh" pod "85a99088-473f-40f5-b963-e3577c99fea4" (UID: "85a99088-473f-40f5-b963-e3577c99fea4"). InnerVolumeSpecName "kube-api-access-rz5wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:12.082564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.082530 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:49:12.145034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.134925 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5wh\" (UniqueName: \"kubernetes.io/projected/85a99088-473f-40f5-b963-e3577c99fea4-kube-api-access-rz5wh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.145034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.134965 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a99088-473f-40f5-b963-e3577c99fea4-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.235994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.235636 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz\" (UniqueName: \"kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz\") pod \"4779d85c-759c-464f-b860-c28044707aa6\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " Nov 03 19:49:12.235994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.235893 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume\") pod \"4779d85c-759c-464f-b860-c28044707aa6\" (UID: \"4779d85c-759c-464f-b860-c28044707aa6\") " Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.237287 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume" pod "4779d85c-759c-464f-b860-c28044707aa6" (UID: "4779d85c-759c-464f-b860-c28044707aa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.240793 5233 generic.go:356] "Generic (PLEG): container finished" podID="85a99088-473f-40f5-b963-e3577c99fea4" containerID="e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a" exitCode=0 Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.241022 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" event={"ID":"85a99088-473f-40f5-b963-e3577c99fea4","Type":"ContainerDied","Data":"e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a"} Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.241078 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" event={"ID":"85a99088-473f-40f5-b963-e3577c99fea4","Type":"ContainerDied","Data":"f5bf8b9eaf6eefe3dc65360aea58e451e01a43f4acae4e5bcf78028dac63529f"} Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.241110 5233 scope.go:122] "RemoveContainer" containerID="e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a" Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.241289 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h" Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.249835 5233 generic.go:356] "Generic (PLEG): container finished" podID="4779d85c-759c-464f-b860-c28044707aa6" containerID="2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0" exitCode=0 Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.250476 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.250625 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" event={"ID":"4779d85c-759c-464f-b860-c28044707aa6","Type":"ContainerDied","Data":"2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0"} Nov 03 19:49:12.262590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.250686 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf" event={"ID":"4779d85c-759c-464f-b860-c28044707aa6","Type":"ContainerDied","Data":"c5c6394848a665a5d93802a4c3fe05d1eea549ff2ba78075108753b71376030b"} Nov 03 19:49:12.266372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.266337 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz" pod "4779d85c-759c-464f-b860-c28044707aa6" (UID: "4779d85c-759c-464f-b860-c28044707aa6"). InnerVolumeSpecName "kube-api-access-zbrrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:12.328993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.328543 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h"] Nov 03 19:49:12.333981 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.333919 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eae28ff0-848f458b89-n8j6h"] Nov 03 19:49:12.340318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.339394 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrrz\" (UniqueName: \"kubernetes.io/projected/4779d85c-759c-464f-b860-c28044707aa6-kube-api-access-zbrrz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.340318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.339428 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4779d85c-759c-464f-b860-c28044707aa6-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.342728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.342291 5233 scope.go:122] "RemoveContainer" containerID="e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a" Nov 03 19:49:12.342816 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:49:12.342770 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a\": container with ID starting with e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a not found: ID does not exist" containerID="e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a" Nov 03 19:49:12.342872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.342818 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a"} err="failed to get container status \"e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a\": rpc error: code = NotFound desc = could not find container \"e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a\": container with ID starting with e32e860d2f4ff3b04903a31c82a4073f7f79ff93168b34cb1ede0b42948be46a not found: ID does not exist" Nov 03 19:49:12.342872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.342846 5233 scope.go:122] "RemoveContainer" containerID="2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0" Nov 03 19:49:12.361566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.361548 5233 scope.go:122] "RemoveContainer" containerID="2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0" Nov 03 19:49:12.361952 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:49:12.361918 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0\": container with ID starting with 2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0 not found: ID does not exist" containerID="2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0" Nov 03 19:49:12.362086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.361961 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0"} err="failed to get container status \"2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0\": rpc error: code = NotFound desc = could not find container \"2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0\": container with ID starting with 2b246b8fd428337b65b4b56e53ae163ba51f649c11acdf8dbd9806b4b64e4fd0 not found: ID does not exist" Nov 03 19:49:12.475196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.475170 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:12.542321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.542240 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt\" (UniqueName: \"kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt\") pod \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " Nov 03 19:49:12.542468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.542446 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca\") pod \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " Nov 03 19:49:12.542684 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.542657 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca\") pod \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\" (UID: \"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7\") " Nov 03 19:49:12.542837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.542817 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca" pod "ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" (UID: "ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:12.543395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.543374 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.544476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.544455 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt" pod "ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" (UID: "ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7"). InnerVolumeSpecName "kube-api-access-5frzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:12.544722 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.544700 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca" pod "ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" (UID: "ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:12.631009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.624215 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf"] Nov 03 19:49:12.671567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.660334 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.671567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.660371 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5frzt\" (UniqueName: \"kubernetes.io/projected/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7-kube-api-access-5frzt\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.727995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:12.723064 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eae28ff0-579765cf7f-v6fdf"] Nov 03 19:49:13.367044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:13.364327 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" event={"ID":"ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7","Type":"ContainerDied","Data":"1523fb0301bc274b588e7dcdee78452a840fd7a43f4d6a08ad3cdbfbd8b0b449"} Nov 03 19:49:13.367044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:13.364419 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1523fb0301bc274b588e7dcdee78452a840fd7a43f4d6a08ad3cdbfbd8b0b449" Nov 03 19:49:13.367044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:13.364515 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7" Nov 03 19:49:13.839908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:13.839876 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4779d85c-759c-464f-b860-c28044707aa6" path="/var/lib/kubelet/pods/4779d85c-759c-464f-b860-c28044707aa6/volumes" Nov 03 19:49:13.840495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:13.840479 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a99088-473f-40f5-b963-e3577c99fea4" path="/var/lib/kubelet/pods/85a99088-473f-40f5-b963-e3577c99fea4/volumes" Nov 03 19:49:14.988897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:14.988868 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:49:16.135649 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:16.135614 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz"] Nov 03 19:49:16.143229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:16.143198 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae28ff0-l7rjz"] Nov 03 19:49:17.842178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:17.842143 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9" path="/var/lib/kubelet/pods/6a3f8254-5980-4e2f-bf89-bb8ae84d3bc9/volumes" Nov 03 19:49:20.037618 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.037585 5233 scope.go:122] "RemoveContainer" containerID="92dd494d499c386d09e80667b959573b27fbd9d2909708a44ba791d6d57dba9a" Nov 03 19:49:20.049791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.049766 5233 scope.go:122] "RemoveContainer" containerID="6db9159dbc26e161e494c7cfd3456092e09fb48484c279179ea123fe54a8d9c4" Nov 03 19:49:20.061696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.061672 5233 scope.go:122] "RemoveContainer" containerID="767ac36a518179340b8f6c603a746c4298f7e6f496775a405396a20a1b912a6f" Nov 03 19:49:20.075433 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.075414 5233 scope.go:122] "RemoveContainer" containerID="43eb5e0f4574ada090d449eb5fc299849456818862b73da4bf917199bc412a6f" Nov 03 19:49:20.087195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.087171 5233 scope.go:122] "RemoveContainer" containerID="1d13ba10e99558714ac6fae98f0bd3c79ec90875a0d654dc70a5309c4947876b" Nov 03 19:49:20.101728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.101701 5233 scope.go:122] "RemoveContainer" containerID="524c17871f3818e20b52fa7e58f70a2b23e3250dabf90b1710a5cce81afc5601" Nov 03 19:49:20.114376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.114352 5233 scope.go:122] "RemoveContainer" containerID="31ea3755970852343b23961de1a89ce18df3c28e3b6a3b3fcaab77cc82853d6f" Nov 03 19:49:20.125965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.125935 5233 scope.go:122] "RemoveContainer" containerID="fa53696fb44e29ac174be16207b10eefe105105833183e0d21ef6cb7aa567674" Nov 03 19:49:20.137542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.137514 5233 scope.go:122] "RemoveContainer" containerID="52bbe6b56d296dedbb60232f18c226f4f17e1c09ccd3fa073f942571a7076aeb" Nov 03 19:49:20.147959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.147941 5233 scope.go:122] "RemoveContainer" containerID="e4f1cf880974fe159a644e2c78b9ea12f257a354a49cc46f3f9e8b0a18b757bf" Nov 03 19:49:20.161311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.161289 5233 scope.go:122] "RemoveContainer" containerID="5d7eddb4f012b6664c2e1a9ae4cf8432a968a0c9820c9bf1a1c477768f44cb79" Nov 03 19:49:20.175783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.175764 5233 scope.go:122] "RemoveContainer" containerID="ad701871e46c8c5b53ba521760cc978faef67e8b2504fce6576072a1acb9aed9" Nov 03 19:49:20.227705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.227681 5233 scope.go:122] "RemoveContainer" containerID="dc0e861b6d6ff5919ccad77f59f8a778cbbe148da9cd5fe4941b520a4422b0de" Nov 03 19:49:20.240807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.240788 5233 scope.go:122] "RemoveContainer" containerID="4e4332303c3c8ff642083e3f28b73a31a9dc77819f38bfb7c2161bd31df84940" Nov 03 19:49:20.252722 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.252701 5233 scope.go:122] "RemoveContainer" containerID="50ce50c87507f42b515f90dd98e17dbedd1f0866ba46f19826080a152e836435" Nov 03 19:49:20.263726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.263699 5233 scope.go:122] "RemoveContainer" containerID="3cba72afa1ed947df6677931993958fc280098f7cf39df5fa5c454267787d461" Nov 03 19:49:20.943371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.943341 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7"] Nov 03 19:49:20.947994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:20.947958 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f292e03c-lnlp7"] Nov 03 19:49:21.464713 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:49:21.464677 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:49:21.857814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:21.857712 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" path="/var/lib/kubelet/pods/ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7/volumes" Nov 03 19:49:25.791622 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:25.791583 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2"] Nov 03 19:49:25.792485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:25.792451 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" containerID="cri-o://044b140e395aa8c4bd5f16d25d22bf72eea13d33149bdb3a1170ecc3860ccc50" gracePeriod=30 Nov 03 19:49:29.342913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:29.342875 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4"] Nov 03 19:49:29.343432 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:29.343186 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" containerID="cri-o://91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48" gracePeriod=30 Nov 03 19:49:31.011917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.011879 5233 generic.go:356] "Generic (PLEG): container finished" podID="de74f23f-0e40-4ae6-be24-e052f803e237" containerID="044b140e395aa8c4bd5f16d25d22bf72eea13d33149bdb3a1170ecc3860ccc50" exitCode=0 Nov 03 19:49:31.014388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.014355 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" event={"ID":"de74f23f-0e40-4ae6-be24-e052f803e237","Type":"ContainerDied","Data":"044b140e395aa8c4bd5f16d25d22bf72eea13d33149bdb3a1170ecc3860ccc50"} Nov 03 19:49:31.564029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.563985 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:49:31.663890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.663808 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.663890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.663850 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.664185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.663919 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.664185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664134 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:31.664185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664153 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.664391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664232 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664457 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664532 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664549 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664671 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664712 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data\") pod \"de74f23f-0e40-4ae6-be24-e052f803e237\" (UID: \"de74f23f-0e40-4ae6-be24-e052f803e237\") " Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664815 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:31.665067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.664915 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:31.665563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.665070 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:31.665563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.665479 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de74f23f-0e40-4ae6-be24-e052f803e237-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.665563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.665500 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.665563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.665514 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.665563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.665523 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.665563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.665531 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.667404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.667381 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:49:31.667503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.667461 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "kube-api-access-mq6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:31.667643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.667628 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:31.670092 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.670070 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy" pod "de74f23f-0e40-4ae6-be24-e052f803e237" (UID: "de74f23f-0e40-4ae6-be24-e052f803e237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:31.766752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.766718 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de74f23f-0e40-4ae6-be24-e052f803e237-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.766752 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.766750 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6f7\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-kube-api-access-mq6f7\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.767049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.766766 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de74f23f-0e40-4ae6-be24-e052f803e237-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:31.767049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:31.766779 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de74f23f-0e40-4ae6-be24-e052f803e237-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:32.025560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:32.025523 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" event={"ID":"de74f23f-0e40-4ae6-be24-e052f803e237","Type":"ContainerDied","Data":"93b8ac97c53eae01133783c8aba5e4e4fe3caba7a0d2e3cf1b87b7dc5da63bf8"} Nov 03 19:49:32.026145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:32.025596 5233 scope.go:122] "RemoveContainer" containerID="044b140e395aa8c4bd5f16d25d22bf72eea13d33149bdb3a1170ecc3860ccc50" Nov 03 19:49:32.026145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:32.025768 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2" Nov 03 19:49:32.058694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:32.058655 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2"] Nov 03 19:49:32.064172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:32.064148 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-7mzm2"] Nov 03 19:49:33.839788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:33.839755 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" path="/var/lib/kubelet/pods/de74f23f-0e40-4ae6-be24-e052f803e237/volumes" Nov 03 19:49:34.395891 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.395830 5233 patch_prober.go:29] interesting pod/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 19:49:34.396332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.396302 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 19:49:34.691153 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.691118 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:49:34.837365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837333 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837395 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837485 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837601 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837662 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:34.837770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837719 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837754 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837796 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.837930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837893 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:34.837930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.837913 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.838149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.838005 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy\") pod \"19e8ce03-af7c-40bd-9980-d40d0184413f\" (UID: \"19e8ce03-af7c-40bd-9980-d40d0184413f\") " Nov 03 19:49:34.838378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.838353 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:34.838775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.838746 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.838898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.838775 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.838898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.838790 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.841792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.841239 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:34.841792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.841386 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:34.842236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.842115 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:34.842354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.842320 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:49:34.843277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.843252 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "kube-api-access-xl6z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:34.847540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.847514 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo" pod "19e8ce03-af7c-40bd-9980-d40d0184413f" (UID: "19e8ce03-af7c-40bd-9980-d40d0184413f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:49:34.950862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.949775 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.950862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.949828 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.950862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.949847 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e8ce03-af7c-40bd-9980-d40d0184413f-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.950862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.949859 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e8ce03-af7c-40bd-9980-d40d0184413f-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.950862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.949875 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6z7\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-kube-api-access-xl6z7\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.950862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:34.949888 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e8ce03-af7c-40bd-9980-d40d0184413f-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:35.151996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.150504 5233 generic.go:356] "Generic (PLEG): container finished" podID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerID="91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48" exitCode=0 Nov 03 19:49:35.151996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.150585 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" event={"ID":"19e8ce03-af7c-40bd-9980-d40d0184413f","Type":"ContainerDied","Data":"91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48"} Nov 03 19:49:35.151996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.150639 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" event={"ID":"19e8ce03-af7c-40bd-9980-d40d0184413f","Type":"ContainerDied","Data":"ad829c52a892b4b2c64092e32c2d6aa90aa2961bd99cfc07275ab1c63b1aeebd"} Nov 03 19:49:35.151996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.150669 5233 scope.go:122] "RemoveContainer" containerID="91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48" Nov 03 19:49:35.151996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.150848 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4" Nov 03 19:49:35.302083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.302049 5233 scope.go:122] "RemoveContainer" containerID="b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058" Nov 03 19:49:35.325692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.325668 5233 scope.go:122] "RemoveContainer" containerID="91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48" Nov 03 19:49:35.326184 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:49:35.326154 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48\": container with ID starting with 91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48 not found: ID does not exist" containerID="91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48" Nov 03 19:49:35.326278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.326195 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48"} err="failed to get container status \"91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48\": rpc error: code = NotFound desc = could not find container \"91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48\": container with ID starting with 91ab07f4255b06fa53383bbba0b74b39625883ba1e9a83b82c77011c63192f48 not found: ID does not exist" Nov 03 19:49:35.326278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.326220 5233 scope.go:122] "RemoveContainer" containerID="b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058" Nov 03 19:49:35.326576 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:49:35.326551 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058\": container with ID starting with b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058 not found: ID does not exist" containerID="b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058" Nov 03 19:49:35.326650 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.326584 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058"} err="failed to get container status \"b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058\": rpc error: code = NotFound desc = could not find container \"b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058\": container with ID starting with b49369ada0191c020954d7799f073dafd17ca9791979d23b4df8f630a49f6058 not found: ID does not exist" Nov 03 19:49:35.336997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.336952 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4"] Nov 03 19:49:35.342335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.342205 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f292e03c-openshift-default-7bd567666f-zvds4"] Nov 03 19:49:35.865692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:35.865659 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" path="/var/lib/kubelet/pods/19e8ce03-af7c-40bd-9980-d40d0184413f/volumes" Nov 03 19:49:38.085502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.085463 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p"] Nov 03 19:49:38.086562 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086496 5233 cpu_manager.go:414] "containerMap: removing container" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086561 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086614 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4779d85c-759c-464f-b860-c28044707aa6" containerName="payload-pre-processing" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086624 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4779d85c-759c-464f-b860-c28044707aa6" containerName="payload-pre-processing" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086642 5233 cpu_manager.go:414] "containerMap: removing container" podUID="85a99088-473f-40f5-b963-e3577c99fea4" containerName="payload-processing" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086650 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a99088-473f-40f5-b963-e3577c99fea4" containerName="payload-processing" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086675 5233 cpu_manager.go:414] "containerMap: removing container" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086683 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086716 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" containerName="revoke-keys" Nov 03 19:49:38.086728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086724 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" containerName="revoke-keys" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086782 5233 cpu_manager.go:414] "containerMap: removing container" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.086791 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.087040 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.087059 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4779d85c-759c-464f-b860-c28044707aa6" containerName="payload-pre-processing" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.087068 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a99088-473f-40f5-b963-e3577c99fea4" containerName="payload-processing" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.087079 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="de74f23f-0e40-4ae6-be24-e052f803e237" containerName="istio-proxy" Nov 03 19:49:38.087175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.087088 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3abc8b-2acf-41bd-9b11-3d3be8cf4fe7" containerName="revoke-keys" Nov 03 19:49:38.093201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.093162 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.101216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.101193 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8cff38\"" Nov 03 19:49:38.101375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.101291 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8cff38-dockercfg-kf7kk\"" Nov 03 19:49:38.105901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.105871 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p"] Nov 03 19:49:38.117301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.117260 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb"] Nov 03 19:49:38.119412 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.119388 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e8ce03-af7c-40bd-9980-d40d0184413f" containerName="istio-proxy" Nov 03 19:49:38.123320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.123286 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb"] Nov 03 19:49:38.123453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.123388 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.154685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.154652 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6jz\" (UniqueName: \"kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz\") pod \"payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.154915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.154890 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.155077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.155053 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppglk\" (UniqueName: \"kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk\") pod \"payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.155166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.155133 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.257095 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.257055 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppglk\" (UniqueName: \"kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk\") pod \"payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.257288 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.257140 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.257352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.257319 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6jz\" (UniqueName: \"kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz\") pod \"payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.257490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.257471 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.258013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.257967 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.258131 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.258113 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.308811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.308775 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppglk\" (UniqueName: \"kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk\") pod \"payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.334065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.334035 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6jz\" (UniqueName: \"kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz\") pod \"payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.422340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.422254 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:38.482009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.469624 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:38.997638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:38.997603 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p"] Nov 03 19:49:39.008133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.008106 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb"] Nov 03 19:49:39.259036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.258934 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" event={"ID":"78363f8d-d3f4-49ea-aab2-640663ac3596","Type":"ContainerStarted","Data":"14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d"} Nov 03 19:49:39.259406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.259038 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" event={"ID":"78363f8d-d3f4-49ea-aab2-640663ac3596","Type":"ContainerStarted","Data":"65ec19932e9ada18fe423815703e5056987a7fd978b614085b03272f5e626daf"} Nov 03 19:49:39.259406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.259158 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:39.260715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.260691 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" event={"ID":"e354a571-d108-451d-a82e-cf000844b8e1","Type":"ContainerStarted","Data":"85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4"} Nov 03 19:49:39.260822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.260730 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" event={"ID":"e354a571-d108-451d-a82e-cf000844b8e1","Type":"ContainerStarted","Data":"ef26b9e8c3103ddb8ab475ed98200fa593e6fe843fb5e5435ff5c3ca67b96439"} Nov 03 19:49:39.260822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.260817 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:39.287701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.287642 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" podStartSLOduration=1.287624422 podStartE2EDuration="1.287624422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:39.284333634 +0000 UTC m=+1345.625447116" watchObservedRunningTime="2026-11-03 19:49:39.287624422 +0000 UTC m=+1345.628737908" Nov 03 19:49:39.308138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:39.308080 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" podStartSLOduration=1.3080417500000001 podStartE2EDuration="1.30804175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:39.304439306 +0000 UTC m=+1345.645552791" watchObservedRunningTime="2026-11-03 19:49:39.30804175 +0000 UTC m=+1345.649155234" Nov 03 19:49:50.275781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:50.275755 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:49:50.276323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:49:50.275824 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:49:51.279175 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:49:51.279142 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:50:02.876198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.876167 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj"] Nov 03 19:50:02.880750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.880730 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:02.885514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.885492 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:02.885654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.885590 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:50:02.891345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.891320 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj"] Nov 03 19:50:02.966693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.966652 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:02.967208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.967181 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:02.967322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:02.967231 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmjk\" (UniqueName: \"kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.068952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.068912 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.069166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.069046 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.069166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.069093 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmjk\" (UniqueName: \"kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.069758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.069733 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.071506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.071480 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.081640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.081620 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmjk\" (UniqueName: \"kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk\") pod \"maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.213501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.213473 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:03.503770 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:03.503739 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45238c00_498b_4fb4_8f41_62abd7bde4c9.slice/crio-df66f8bff08006f1a87e9f4b88a04206df605765ef4ebe7f21dbf732cc54c00b WatchSource:0}: Error finding container df66f8bff08006f1a87e9f4b88a04206df605765ef4ebe7f21dbf732cc54c00b: Status 404 returned error can't find the container with id df66f8bff08006f1a87e9f4b88a04206df605765ef4ebe7f21dbf732cc54c00b Nov 03 19:50:03.506137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.506109 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj"] Nov 03 19:50:03.942450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.942416 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" event={"ID":"45238c00-498b-4fb4-8f41-62abd7bde4c9","Type":"ContainerStarted","Data":"7abcbb52931e73a4d9699ee4553c5662e5589f19282ed41b483a3e81b2164cee"} Nov 03 19:50:03.942877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.942480 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" event={"ID":"45238c00-498b-4fb4-8f41-62abd7bde4c9","Type":"ContainerStarted","Data":"df66f8bff08006f1a87e9f4b88a04206df605765ef4ebe7f21dbf732cc54c00b"} Nov 03 19:50:03.962935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:03.962891 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" podStartSLOduration=1.9628770869999999 podStartE2EDuration="1.962877087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:03.961849137 +0000 UTC m=+1370.302962619" watchObservedRunningTime="2026-11-03 19:50:03.962877087 +0000 UTC m=+1370.303990572" Nov 03 19:50:04.955698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:04.955666 5233 generic.go:356] "Generic (PLEG): container finished" podID="45238c00-498b-4fb4-8f41-62abd7bde4c9" containerID="7abcbb52931e73a4d9699ee4553c5662e5589f19282ed41b483a3e81b2164cee" exitCode=0 Nov 03 19:50:04.956112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:04.955762 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" event={"ID":"45238c00-498b-4fb4-8f41-62abd7bde4c9","Type":"ContainerDied","Data":"7abcbb52931e73a4d9699ee4553c5662e5589f19282ed41b483a3e81b2164cee"} Nov 03 19:50:06.284232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.284206 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:06.351794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.351703 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca\") pod \"45238c00-498b-4fb4-8f41-62abd7bde4c9\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " Nov 03 19:50:06.351966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.351827 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca\") pod \"45238c00-498b-4fb4-8f41-62abd7bde4c9\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " Nov 03 19:50:06.351966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.351940 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk\" (UniqueName: \"kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk\") pod \"45238c00-498b-4fb4-8f41-62abd7bde4c9\" (UID: \"45238c00-498b-4fb4-8f41-62abd7bde4c9\") " Nov 03 19:50:06.352204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.352176 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca" pod "45238c00-498b-4fb4-8f41-62abd7bde4c9" (UID: "45238c00-498b-4fb4-8f41-62abd7bde4c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:06.352700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.352674 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:06.354059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.353945 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca" pod "45238c00-498b-4fb4-8f41-62abd7bde4c9" (UID: "45238c00-498b-4fb4-8f41-62abd7bde4c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:06.354220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.354198 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk" pod "45238c00-498b-4fb4-8f41-62abd7bde4c9" (UID: "45238c00-498b-4fb4-8f41-62abd7bde4c9"). InnerVolumeSpecName "kube-api-access-vcmjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:06.454778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.454742 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45238c00-498b-4fb4-8f41-62abd7bde4c9-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:06.454778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:06.454777 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmjk\" (UniqueName: \"kubernetes.io/projected/45238c00-498b-4fb4-8f41-62abd7bde4c9-kube-api-access-vcmjk\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:07.076257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:07.076227 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" Nov 03 19:50:07.076257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:07.076241 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj" event={"ID":"45238c00-498b-4fb4-8f41-62abd7bde4c9","Type":"ContainerDied","Data":"df66f8bff08006f1a87e9f4b88a04206df605765ef4ebe7f21dbf732cc54c00b"} Nov 03 19:50:07.076465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:07.076289 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df66f8bff08006f1a87e9f4b88a04206df605765ef4ebe7f21dbf732cc54c00b" Nov 03 19:50:07.637995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:07.637819 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh"] Nov 03 19:50:07.644383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:07.644352 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5"] Nov 03 19:50:07.649829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:07.649801 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446"] Nov 03 19:50:13.016630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.014307 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k"] Nov 03 19:50:13.029162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.029131 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n"] Nov 03 19:50:13.850950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.850919 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9"] Nov 03 19:50:13.851789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.851770 5233 cpu_manager.go:414] "containerMap: removing container" podUID="45238c00-498b-4fb4-8f41-62abd7bde4c9" containerName="revoke-keys" Nov 03 19:50:13.851883 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.851791 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45238c00-498b-4fb4-8f41-62abd7bde4c9" containerName="revoke-keys" Nov 03 19:50:13.851958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.851946 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="45238c00-498b-4fb4-8f41-62abd7bde4c9" containerName="revoke-keys" Nov 03 19:50:13.857885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.857862 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:13.863913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.863884 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9"] Nov 03 19:50:13.927321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.927295 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:13.927565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:13.927546 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:50:14.000948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.000911 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v465c\" (UniqueName: \"kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.001152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.001098 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.001298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.001278 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.103453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.103364 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.103892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.103587 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v465c\" (UniqueName: \"kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.103892 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.103724 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.104200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.104168 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.105965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.105937 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.116082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.116055 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v465c\" (UniqueName: \"kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c\") pod \"maas-api-revoke-keys-e2e-api-a-10e551-dtfs9\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.191910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.191882 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:14.485365 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:14.485323 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee55c16_a923_473a_8e62_d2b025b0baa5.slice/crio-c737c1b074e5e3f8a5ad943f96aaa8cc2304c06dad55ddcb1d375f067a804d36 WatchSource:0}: Error finding container c737c1b074e5e3f8a5ad943f96aaa8cc2304c06dad55ddcb1d375f067a804d36: Status 404 returned error can't find the container with id c737c1b074e5e3f8a5ad943f96aaa8cc2304c06dad55ddcb1d375f067a804d36 Nov 03 19:50:14.486061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:14.486038 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9"] Nov 03 19:50:15.244475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:15.244440 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" event={"ID":"bee55c16-a923-473a-8e62-d2b025b0baa5","Type":"ContainerStarted","Data":"1d7a33fa137b0a35eb891d84edd0bd36e18ef337b50b44196798461bf487533e"} Nov 03 19:50:15.244929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:15.244524 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" event={"ID":"bee55c16-a923-473a-8e62-d2b025b0baa5","Type":"ContainerStarted","Data":"c737c1b074e5e3f8a5ad943f96aaa8cc2304c06dad55ddcb1d375f067a804d36"} Nov 03 19:50:15.266792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:15.266746 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" podStartSLOduration=2.266732935 podStartE2EDuration="2.266732935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:15.262217519 +0000 UTC m=+1381.603331028" watchObservedRunningTime="2026-11-03 19:50:15.266732935 +0000 UTC m=+1381.607846474" Nov 03 19:50:16.255220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:16.255189 5233 generic.go:356] "Generic (PLEG): container finished" podID="bee55c16-a923-473a-8e62-d2b025b0baa5" containerID="1d7a33fa137b0a35eb891d84edd0bd36e18ef337b50b44196798461bf487533e" exitCode=0 Nov 03 19:50:16.255707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:16.255232 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" event={"ID":"bee55c16-a923-473a-8e62-d2b025b0baa5","Type":"ContainerDied","Data":"1d7a33fa137b0a35eb891d84edd0bd36e18ef337b50b44196798461bf487533e"} Nov 03 19:50:17.485323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.484952 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:17.514829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.514788 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c\" (UniqueName: \"kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c\") pod \"bee55c16-a923-473a-8e62-d2b025b0baa5\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " Nov 03 19:50:17.515026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.514843 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca\") pod \"bee55c16-a923-473a-8e62-d2b025b0baa5\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " Nov 03 19:50:17.515107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.515034 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca\") pod \"bee55c16-a923-473a-8e62-d2b025b0baa5\" (UID: \"bee55c16-a923-473a-8e62-d2b025b0baa5\") " Nov 03 19:50:17.516545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.516517 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca" pod "bee55c16-a923-473a-8e62-d2b025b0baa5" (UID: "bee55c16-a923-473a-8e62-d2b025b0baa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:17.522816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.521811 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c" pod "bee55c16-a923-473a-8e62-d2b025b0baa5" (UID: "bee55c16-a923-473a-8e62-d2b025b0baa5"). InnerVolumeSpecName "kube-api-access-v465c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:17.522816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.522203 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca" pod "bee55c16-a923-473a-8e62-d2b025b0baa5" (UID: "bee55c16-a923-473a-8e62-d2b025b0baa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:17.617010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.616375 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq"] Nov 03 19:50:17.617916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.617803 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v465c\" (UniqueName: \"kubernetes.io/projected/bee55c16-a923-473a-8e62-d2b025b0baa5-kube-api-access-v465c\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:17.618077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.617923 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:17.618077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.617939 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee55c16-a923-473a-8e62-d2b025b0baa5-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:17.619658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.619619 5233 cpu_manager.go:414] "containerMap: removing container" podUID="bee55c16-a923-473a-8e62-d2b025b0baa5" containerName="revoke-keys" Nov 03 19:50:17.619658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.619645 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee55c16-a923-473a-8e62-d2b025b0baa5" containerName="revoke-keys" Nov 03 19:50:17.619872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.619828 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee55c16-a923-473a-8e62-d2b025b0baa5" containerName="revoke-keys" Nov 03 19:50:17.647557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.641735 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq"] Nov 03 19:50:17.647557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.641858 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:17.740057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.736591 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ab584a-dockercfg-mz46b\"" Nov 03 19:50:17.740057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:17.736708 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ab584a\"" Nov 03 19:50:18.048017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.041227 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" containerName="payload-pre-processing" containerID="cri-o://bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16" gracePeriod=25 Nov 03 19:50:18.048017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.041691 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" podUID="62f81783-cd80-4436-bc8e-d9870342432e" containerName="payload-processing" containerID="cri-o://9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa" gracePeriod=25 Nov 03 19:50:18.048017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.042642 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwbg\" (UniqueName: \"kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg\") pod \"payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.048017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.043224 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.170014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.162192 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.171368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.168964 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.175676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.175639 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwbg\" (UniqueName: \"kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg\") pod \"payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.259770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.255827 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwbg\" (UniqueName: \"kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg\") pod \"payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.464036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.459745 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" event={"ID":"bee55c16-a923-473a-8e62-d2b025b0baa5","Type":"ContainerDied","Data":"c737c1b074e5e3f8a5ad943f96aaa8cc2304c06dad55ddcb1d375f067a804d36"} Nov 03 19:50:18.464036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.459814 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c737c1b074e5e3f8a5ad943f96aaa8cc2304c06dad55ddcb1d375f067a804d36" Nov 03 19:50:18.464036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.459884 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9" Nov 03 19:50:18.464036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.461319 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:18.833671 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:18.832094 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq"] Nov 03 19:50:18.852028 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:18.851200 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5318a5a_001a_4e66_a71c_a976de821d56.slice/crio-5329a343e8ef55b3fedf8e233b630b04248aac221e6bb872b886e1bb7319a622 WatchSource:0}: Error finding container 5329a343e8ef55b3fedf8e233b630b04248aac221e6bb872b886e1bb7319a622: Status 404 returned error can't find the container with id 5329a343e8ef55b3fedf8e233b630b04248aac221e6bb872b886e1bb7319a622 Nov 03 19:50:19.471009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:19.470954 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" event={"ID":"b5318a5a-001a-4e66-a71c-a976de821d56","Type":"ContainerStarted","Data":"ac7f5cbf04522ab9fb6f87a2a2fc28c7203a76390a369ee9899f37d1a5b75afe"} Nov 03 19:50:19.471188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:19.471028 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:19.471188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:19.471045 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" event={"ID":"b5318a5a-001a-4e66-a71c-a976de821d56","Type":"ContainerStarted","Data":"5329a343e8ef55b3fedf8e233b630b04248aac221e6bb872b886e1bb7319a622"} Nov 03 19:50:19.495016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:19.494943 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" podStartSLOduration=2.494923914 podStartE2EDuration="2.494923914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:19.49202835 +0000 UTC m=+1385.833141835" watchObservedRunningTime="2026-11-03 19:50:19.494923914 +0000 UTC m=+1385.836037399" Nov 03 19:50:20.857055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.857015 5233 scope.go:122] "RemoveContainer" containerID="e0ea5b971b5706bc04f72cfa8f0ff1a5dc35c42544b03d60f4ad0437e4d16011" Nov 03 19:50:20.874960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.874933 5233 scope.go:122] "RemoveContainer" containerID="5ba7d7574c558e0ff8a7f6c26a4be0933262fd714a4e4126296309cf6244254f" Nov 03 19:50:20.889955 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.889932 5233 scope.go:122] "RemoveContainer" containerID="c63cb1c75fe575bae745df8dcd2b4fdd6bbc9edcb84d410d0da5008db19fe30a" Nov 03 19:50:20.906254 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.906230 5233 scope.go:122] "RemoveContainer" containerID="c7cd285058234a2d165cca2c7eed52154cc61769d696b1d2bea89ef689182809" Nov 03 19:50:20.921901 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.921875 5233 scope.go:122] "RemoveContainer" containerID="982c120d31c4f2bc4ab54feb5a0499c9aa3cded01298eaa711982588b843df99" Nov 03 19:50:20.936517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.936488 5233 scope.go:122] "RemoveContainer" containerID="12276ab1a6c7eda660e67a42dcf95b1522cbc4a9975d7c1e17dee17a30ffac5f" Nov 03 19:50:20.955148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.955119 5233 scope.go:122] "RemoveContainer" containerID="5a41b0acab7ff4b93d2361c084eadc0454d9fcaabe57efff3a6b2aca2ab6ff8d" Nov 03 19:50:20.962075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.962051 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm"] Nov 03 19:50:20.968640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.968618 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5"] Nov 03 19:50:20.968774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.968753 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:20.973528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.973502 5233 scope.go:122] "RemoveContainer" containerID="c87a718f05802219d91345f77b722f5a60b89f8df11640249c19c433fa9e3ae5" Nov 03 19:50:20.974876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.974851 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:50:20.976792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:20.976770 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:21.014208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.014175 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm"] Nov 03 19:50:21.014368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.014278 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.025436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.025343 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5"] Nov 03 19:50:21.047249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.047206 5233 scope.go:122] "RemoveContainer" containerID="790a1450a4eea90713b7053e10170f474e3673755a1073b8b2f862c234cdc017" Nov 03 19:50:21.063646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.063625 5233 scope.go:122] "RemoveContainer" containerID="17401bf2bd8261b4f2ca0db461804675cb02b511e4155c405dfec6625c2d7e36" Nov 03 19:50:21.110722 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.110690 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnxh\" (UniqueName: \"kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.110842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.110822 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.111046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.111021 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.111149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.111114 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.111230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.111213 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.111315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.111268 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfkn\" (UniqueName: \"kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.212878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.212848 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.213068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.212918 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.213068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.212965 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.213193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.213128 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfkn\" (UniqueName: \"kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.213418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.213388 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnxh\" (UniqueName: \"kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.213555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.213511 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.213709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.213687 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.213792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.213738 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.279995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.267698 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.279995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.274387 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.279995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.275848 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnxh\" (UniqueName: \"kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh\") pod \"maas-api-revoke-sub-ce94f975-8xgz5\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.283643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.282385 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfkn\" (UniqueName: \"kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn\") pod \"maas-api-revoke-sub-087ec59d-t2bxm\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.313604 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.313255 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:21.357415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.357140 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:21.384009 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:21.383497 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:50:21.694029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.693901 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5"] Nov 03 19:50:21.708332 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:21.708288 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e86009_8112_45b6_a16a_854cedb6d0ee.slice/crio-6d091ff0bfaaa0d9e00bdf279acbf1550e9179be61e08cd71e56d8b66febf794 WatchSource:0}: Error finding container 6d091ff0bfaaa0d9e00bdf279acbf1550e9179be61e08cd71e56d8b66febf794: Status 404 returned error can't find the container with id 6d091ff0bfaaa0d9e00bdf279acbf1550e9179be61e08cd71e56d8b66febf794 Nov 03 19:50:21.709221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:21.709079 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm"] Nov 03 19:50:21.713127 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:21.713095 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee3b83c_1446_4a17_b3ca_a4086b670c5a.slice/crio-d0cb56d55e579d41d31d8a9fe1c24cf27bf868493138201da09f1ad43965191c WatchSource:0}: Error finding container d0cb56d55e579d41d31d8a9fe1c24cf27bf868493138201da09f1ad43965191c: Status 404 returned error can't find the container with id d0cb56d55e579d41d31d8a9fe1c24cf27bf868493138201da09f1ad43965191c Nov 03 19:50:22.547799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.547684 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" event={"ID":"d4e86009-8112-45b6-a16a-854cedb6d0ee","Type":"ContainerStarted","Data":"83e9228725fef8f6fbb63e1916cdad603c380fd63c9ba101e2b215353160291e"} Nov 03 19:50:22.547799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.547745 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" event={"ID":"d4e86009-8112-45b6-a16a-854cedb6d0ee","Type":"ContainerStarted","Data":"6d091ff0bfaaa0d9e00bdf279acbf1550e9179be61e08cd71e56d8b66febf794"} Nov 03 19:50:22.549664 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.549632 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" event={"ID":"6ee3b83c-1446-4a17-b3ca-a4086b670c5a","Type":"ContainerStarted","Data":"f5541320c1f77fa767c09ce3fe5de965727e2df7ef0f5dfe1f6f06085b67054f"} Nov 03 19:50:22.549794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.549686 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" event={"ID":"6ee3b83c-1446-4a17-b3ca-a4086b670c5a","Type":"ContainerStarted","Data":"d0cb56d55e579d41d31d8a9fe1c24cf27bf868493138201da09f1ad43965191c"} Nov 03 19:50:22.577050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.576990 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" podStartSLOduration=2.57695407 podStartE2EDuration="2.57695407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:22.567420591 +0000 UTC m=+1388.908534101" watchObservedRunningTime="2026-11-03 19:50:22.57695407 +0000 UTC m=+1388.918067556" Nov 03 19:50:22.602083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.602025 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" podStartSLOduration=2.601997064 podStartE2EDuration="2.601997064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:22.588676009 +0000 UTC m=+1388.929789516" watchObservedRunningTime="2026-11-03 19:50:22.601997064 +0000 UTC m=+1388.943110532" Nov 03 19:50:22.680497 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.680459 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" containerID="cri-o://e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649" gracePeriod=45 Nov 03 19:50:22.680688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.680598 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" containerID="cri-o://7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4" gracePeriod=45 Nov 03 19:50:22.680688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.680599 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" containerID="cri-o://138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b" gracePeriod=45 Nov 03 19:50:22.689388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.689117 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused" Nov 03 19:50:22.790317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.790272 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Nov 03 19:50:22.848920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:22.848803 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": dial tcp 10.134.0.94:8000: connect: connection refused" Nov 03 19:50:23.118251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.118224 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:50:23.120282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.120254 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:50:23.127620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.127596 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:50:23.200920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.200881 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache\") pod \"294e8459-7193-46ec-81b9-5a1e0822eb92\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " Nov 03 19:50:23.201122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.200986 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm\") pod \"294e8459-7193-46ec-81b9-5a1e0822eb92\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " Nov 03 19:50:23.201122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.201024 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home\") pod \"294e8459-7193-46ec-81b9-5a1e0822eb92\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " Nov 03 19:50:23.201238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.201120 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs\" (UniqueName: \"kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs\") pod \"294e8459-7193-46ec-81b9-5a1e0822eb92\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " Nov 03 19:50:23.201238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.201227 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir\") pod \"294e8459-7193-46ec-81b9-5a1e0822eb92\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " Nov 03 19:50:23.201357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.201323 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2\" (UniqueName: \"kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2\") pod \"294e8459-7193-46ec-81b9-5a1e0822eb92\" (UID: \"294e8459-7193-46ec-81b9-5a1e0822eb92\") " Nov 03 19:50:23.202483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.202439 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache" pod "294e8459-7193-46ec-81b9-5a1e0822eb92" (UID: "294e8459-7193-46ec-81b9-5a1e0822eb92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.203626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.203587 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir" pod "294e8459-7193-46ec-81b9-5a1e0822eb92" (UID: "294e8459-7193-46ec-81b9-5a1e0822eb92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.203626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.203600 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home" pod "294e8459-7193-46ec-81b9-5a1e0822eb92" (UID: "294e8459-7193-46ec-81b9-5a1e0822eb92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.205879 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.205847 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2" pod "294e8459-7193-46ec-81b9-5a1e0822eb92" (UID: "294e8459-7193-46ec-81b9-5a1e0822eb92"). InnerVolumeSpecName "kube-api-access-whvg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:23.206722 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.206695 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm" pod "294e8459-7193-46ec-81b9-5a1e0822eb92" (UID: "294e8459-7193-46ec-81b9-5a1e0822eb92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.207998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.207965 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs" pod "294e8459-7193-46ec-81b9-5a1e0822eb92" (UID: "294e8459-7193-46ec-81b9-5a1e0822eb92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:23.304357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304327 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache\") pod \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " Nov 03 19:50:23.304557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304433 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm\") pod \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " Nov 03 19:50:23.304557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304500 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm\") pod \"d199580a-2d32-4e95-bb54-efe923ce3397\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " Nov 03 19:50:23.304557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304540 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg\" (UniqueName: \"kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg\") pod \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " Nov 03 19:50:23.304727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304568 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home\") pod \"d199580a-2d32-4e95-bb54-efe923ce3397\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " Nov 03 19:50:23.304727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304626 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache" pod "b328621b-8a79-4fe3-8c9e-2ef238b93a14" (UID: "b328621b-8a79-4fe3-8c9e-2ef238b93a14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.304727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304638 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir\") pod \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " Nov 03 19:50:23.304948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304751 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl\" (UniqueName: \"kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl\") pod \"d199580a-2d32-4e95-bb54-efe923ce3397\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " Nov 03 19:50:23.304948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304839 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir" pod "b328621b-8a79-4fe3-8c9e-2ef238b93a14" (UID: "b328621b-8a79-4fe3-8c9e-2ef238b93a14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.304948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304889 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home\") pod \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " Nov 03 19:50:23.304948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.304932 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs\" (UniqueName: \"kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs\") pod \"d199580a-2d32-4e95-bb54-efe923ce3397\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " Nov 03 19:50:23.305235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305010 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home" pod "d199580a-2d32-4e95-bb54-efe923ce3397" (UID: "d199580a-2d32-4e95-bb54-efe923ce3397"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.305235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305052 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache\") pod \"d199580a-2d32-4e95-bb54-efe923ce3397\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " Nov 03 19:50:23.305235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305093 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir\") pod \"d199580a-2d32-4e95-bb54-efe923ce3397\" (UID: \"d199580a-2d32-4e95-bb54-efe923ce3397\") " Nov 03 19:50:23.305235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305149 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs\" (UniqueName: \"kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs\") pod \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\" (UID: \"b328621b-8a79-4fe3-8c9e-2ef238b93a14\") " Nov 03 19:50:23.305235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305147 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home" pod "b328621b-8a79-4fe3-8c9e-2ef238b93a14" (UID: "b328621b-8a79-4fe3-8c9e-2ef238b93a14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.305495 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305370 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache" pod "d199580a-2d32-4e95-bb54-efe923ce3397" (UID: "d199580a-2d32-4e95-bb54-efe923ce3397"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.305573 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.305551 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir" pod "d199580a-2d32-4e95-bb54-efe923ce3397" (UID: "d199580a-2d32-4e95-bb54-efe923ce3397"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.306143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306119 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306147 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306164 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306178 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306191 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306205 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294e8459-7193-46ec-81b9-5a1e0822eb92-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306219 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294e8459-7193-46ec-81b9-5a1e0822eb92-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306233 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306247 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvg2\" (UniqueName: \"kubernetes.io/projected/294e8459-7193-46ec-81b9-5a1e0822eb92-kube-api-access-whvg2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306262 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306276 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.306799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.306290 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.307429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.307352 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm" pod "d199580a-2d32-4e95-bb54-efe923ce3397" (UID: "d199580a-2d32-4e95-bb54-efe923ce3397"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.307531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.307453 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg" pod "b328621b-8a79-4fe3-8c9e-2ef238b93a14" (UID: "b328621b-8a79-4fe3-8c9e-2ef238b93a14"). InnerVolumeSpecName "kube-api-access-lvmsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:23.307531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.307490 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl" pod "d199580a-2d32-4e95-bb54-efe923ce3397" (UID: "d199580a-2d32-4e95-bb54-efe923ce3397"). InnerVolumeSpecName "kube-api-access-22jwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:23.307923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.307894 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs" pod "b328621b-8a79-4fe3-8c9e-2ef238b93a14" (UID: "b328621b-8a79-4fe3-8c9e-2ef238b93a14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:23.308008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.307927 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs" pod "d199580a-2d32-4e95-bb54-efe923ce3397" (UID: "d199580a-2d32-4e95-bb54-efe923ce3397"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:23.308440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.308418 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm" pod "b328621b-8a79-4fe3-8c9e-2ef238b93a14" (UID: "b328621b-8a79-4fe3-8c9e-2ef238b93a14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:23.409421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.409282 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d199580a-2d32-4e95-bb54-efe923ce3397-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.409421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.409319 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b328621b-8a79-4fe3-8c9e-2ef238b93a14-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.409421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.409336 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b328621b-8a79-4fe3-8c9e-2ef238b93a14-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.409421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.409350 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d199580a-2d32-4e95-bb54-efe923ce3397-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.409421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.409364 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmsg\" (UniqueName: \"kubernetes.io/projected/b328621b-8a79-4fe3-8c9e-2ef238b93a14-kube-api-access-lvmsg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.409421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.409380 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jwl\" (UniqueName: \"kubernetes.io/projected/d199580a-2d32-4e95-bb54-efe923ce3397-kube-api-access-22jwl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:23.561934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.561892 5233 generic.go:356] "Generic (PLEG): container finished" podID="d4e86009-8112-45b6-a16a-854cedb6d0ee" containerID="83e9228725fef8f6fbb63e1916cdad603c380fd63c9ba101e2b215353160291e" exitCode=0 Nov 03 19:50:23.562436 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.561936 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" event={"ID":"d4e86009-8112-45b6-a16a-854cedb6d0ee","Type":"ContainerDied","Data":"83e9228725fef8f6fbb63e1916cdad603c380fd63c9ba101e2b215353160291e"} Nov 03 19:50:23.563511 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.563486 5233 generic.go:356] "Generic (PLEG): container finished" podID="d199580a-2d32-4e95-bb54-efe923ce3397" containerID="138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b" exitCode=0 Nov 03 19:50:23.563638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.563537 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" Nov 03 19:50:23.563638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.563572 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" event={"ID":"d199580a-2d32-4e95-bb54-efe923ce3397","Type":"ContainerDied","Data":"138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b"} Nov 03 19:50:23.563638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.563610 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh" event={"ID":"d199580a-2d32-4e95-bb54-efe923ce3397","Type":"ContainerDied","Data":"71098cc85171e825d791c435d177d77974047971892b4e8dabfb51e0b7de2f6c"} Nov 03 19:50:23.563638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.563630 5233 scope.go:122] "RemoveContainer" containerID="138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b" Nov 03 19:50:23.565652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.565628 5233 generic.go:356] "Generic (PLEG): container finished" podID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerID="e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649" exitCode=0 Nov 03 19:50:23.565865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.565712 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" event={"ID":"b328621b-8a79-4fe3-8c9e-2ef238b93a14","Type":"ContainerDied","Data":"e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649"} Nov 03 19:50:23.565951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.565883 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" event={"ID":"b328621b-8a79-4fe3-8c9e-2ef238b93a14","Type":"ContainerDied","Data":"093bac7f2223601e168efc439249d8997218272a77bfb46c4df38a0ae40222a9"} Nov 03 19:50:23.565951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.565736 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446" Nov 03 19:50:23.569020 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.568914 5233 generic.go:356] "Generic (PLEG): container finished" podID="6ee3b83c-1446-4a17-b3ca-a4086b670c5a" containerID="f5541320c1f77fa767c09ce3fe5de965727e2df7ef0f5dfe1f6f06085b67054f" exitCode=0 Nov 03 19:50:23.569148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.569053 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" event={"ID":"6ee3b83c-1446-4a17-b3ca-a4086b670c5a","Type":"ContainerDied","Data":"f5541320c1f77fa767c09ce3fe5de965727e2df7ef0f5dfe1f6f06085b67054f"} Nov 03 19:50:23.570917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.570872 5233 generic.go:356] "Generic (PLEG): container finished" podID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerID="7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4" exitCode=0 Nov 03 19:50:23.571083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.570954 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" event={"ID":"294e8459-7193-46ec-81b9-5a1e0822eb92","Type":"ContainerDied","Data":"7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4"} Nov 03 19:50:23.571083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.571034 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" event={"ID":"294e8459-7193-46ec-81b9-5a1e0822eb92","Type":"ContainerDied","Data":"f6e2deb28cd2b322a8d0622ecc4bbb4f2b8ca21e22652b19eef2ac77b3f2328a"} Nov 03 19:50:23.571083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.571067 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5" Nov 03 19:50:23.584466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.584444 5233 scope.go:122] "RemoveContainer" containerID="138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b" Nov 03 19:50:23.584921 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:23.584892 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b\": container with ID starting with 138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b not found: ID does not exist" containerID="138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b" Nov 03 19:50:23.585099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.584926 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b"} err="failed to get container status \"138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b\": rpc error: code = NotFound desc = could not find container \"138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b\": container with ID starting with 138af798fa0138b7c07a31cb9cd9a2d9ebae58928a0a04b46ec7d0291eb7ab7b not found: ID does not exist" Nov 03 19:50:23.585099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.584949 5233 scope.go:122] "RemoveContainer" containerID="e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649" Nov 03 19:50:23.604017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.604001 5233 scope.go:122] "RemoveContainer" containerID="e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649" Nov 03 19:50:23.604333 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:23.604307 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649\": container with ID starting with e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649 not found: ID does not exist" containerID="e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649" Nov 03 19:50:23.604390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.604340 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649"} err="failed to get container status \"e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649\": rpc error: code = NotFound desc = could not find container \"e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649\": container with ID starting with e01d52a21375f0b7a8d5ec4da23d64b01b94db41cfedcbc137ba77f4bc5e5649 not found: ID does not exist" Nov 03 19:50:23.604390 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.604357 5233 scope.go:122] "RemoveContainer" containerID="7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4" Nov 03 19:50:23.617567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.617545 5233 scope.go:122] "RemoveContainer" containerID="7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4" Nov 03 19:50:23.617904 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:23.617878 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4\": container with ID starting with 7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4 not found: ID does not exist" containerID="7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4" Nov 03 19:50:23.618093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.618058 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4"} err="failed to get container status \"7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4\": rpc error: code = NotFound desc = could not find container \"7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4\": container with ID starting with 7b3ff90361580701edd774901f32fcb6b2ef347d4c42186f0845eee43d7231a4 not found: ID does not exist" Nov 03 19:50:23.642105 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.642079 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5"] Nov 03 19:50:23.665410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.665274 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/premium-w6-c5f6cb-kserve-77dffb797f-tw5c5"] Nov 03 19:50:23.669475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.669450 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh"] Nov 03 19:50:23.704273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.704233 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/distinct-w6-c5f6cb-kserve-5794dcf87d-hnsdh"] Nov 03 19:50:23.713653 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.713622 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446"] Nov 03 19:50:23.719037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.719009 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c5f6cb/simulator-w6-c5f6cb-kserve-f59897f86-9j446"] Nov 03 19:50:23.840110 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.840078 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" path="/var/lib/kubelet/pods/294e8459-7193-46ec-81b9-5a1e0822eb92/volumes" Nov 03 19:50:23.840549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.840533 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" path="/var/lib/kubelet/pods/b328621b-8a79-4fe3-8c9e-2ef238b93a14/volumes" Nov 03 19:50:23.841012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.840957 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" path="/var/lib/kubelet/pods/d199580a-2d32-4e95-bb54-efe923ce3397/volumes" Nov 03 19:50:23.976762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:23.976727 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc"] Nov 03 19:50:24.943246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:24.943224 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:24.945814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:24.945794 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:25.087995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.080641 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh\" (UniqueName: \"kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh\") pod \"d4e86009-8112-45b6-a16a-854cedb6d0ee\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " Nov 03 19:50:25.087995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.080733 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca\") pod \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " Nov 03 19:50:25.087995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.080874 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca\") pod \"d4e86009-8112-45b6-a16a-854cedb6d0ee\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " Nov 03 19:50:25.087995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.080953 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca\") pod \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " Nov 03 19:50:25.087995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.081093 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca\") pod \"d4e86009-8112-45b6-a16a-854cedb6d0ee\" (UID: \"d4e86009-8112-45b6-a16a-854cedb6d0ee\") " Nov 03 19:50:25.087995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.081170 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn\" (UniqueName: \"kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn\") pod \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\" (UID: \"6ee3b83c-1446-4a17-b3ca-a4086b670c5a\") " Nov 03 19:50:25.088778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.088488 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca" pod "d4e86009-8112-45b6-a16a-854cedb6d0ee" (UID: "d4e86009-8112-45b6-a16a-854cedb6d0ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:25.088778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.088744 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca" pod "6ee3b83c-1446-4a17-b3ca-a4086b670c5a" (UID: "6ee3b83c-1446-4a17-b3ca-a4086b670c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:25.114420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.111330 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh" pod "d4e86009-8112-45b6-a16a-854cedb6d0ee" (UID: "d4e86009-8112-45b6-a16a-854cedb6d0ee"). InnerVolumeSpecName "kube-api-access-hqnxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:25.115085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.114650 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca" pod "6ee3b83c-1446-4a17-b3ca-a4086b670c5a" (UID: "6ee3b83c-1446-4a17-b3ca-a4086b670c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:25.124174 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.124110 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca" pod "d4e86009-8112-45b6-a16a-854cedb6d0ee" (UID: "d4e86009-8112-45b6-a16a-854cedb6d0ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:25.130029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.127511 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn" pod "6ee3b83c-1446-4a17-b3ca-a4086b670c5a" (UID: "6ee3b83c-1446-4a17-b3ca-a4086b670c5a"). InnerVolumeSpecName "kube-api-access-lnfkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:25.183839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.183738 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqnxh\" (UniqueName: \"kubernetes.io/projected/d4e86009-8112-45b6-a16a-854cedb6d0ee-kube-api-access-hqnxh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:25.183839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.183776 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:25.183839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.183792 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:25.183839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.183806 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:25.183839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.183820 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e86009-8112-45b6-a16a-854cedb6d0ee-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:25.183839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.183834 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfkn\" (UniqueName: \"kubernetes.io/projected/6ee3b83c-1446-4a17-b3ca-a4086b670c5a-kube-api-access-lnfkn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:25.656582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.656550 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" Nov 03 19:50:25.656582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.656573 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm" event={"ID":"6ee3b83c-1446-4a17-b3ca-a4086b670c5a","Type":"ContainerDied","Data":"d0cb56d55e579d41d31d8a9fe1c24cf27bf868493138201da09f1ad43965191c"} Nov 03 19:50:25.656822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.656626 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0cb56d55e579d41d31d8a9fe1c24cf27bf868493138201da09f1ad43965191c" Nov 03 19:50:25.658267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.658245 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" Nov 03 19:50:25.658893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.658305 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5" event={"ID":"d4e86009-8112-45b6-a16a-854cedb6d0ee","Type":"ContainerDied","Data":"6d091ff0bfaaa0d9e00bdf279acbf1550e9179be61e08cd71e56d8b66febf794"} Nov 03 19:50:25.658893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:25.658353 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d091ff0bfaaa0d9e00bdf279acbf1550e9179be61e08cd71e56d8b66febf794" Nov 03 19:50:26.106109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:26.106071 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:50:27.148087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:27.143100 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" podUID="62f81783-cd80-4436-bc8e-d9870342432e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:50:28.978033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:28.977991 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" podUID="7a61bca2-e418-4af1-a018-3b377295f94e" containerName="payload-processing" containerID="cri-o://1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2" gracePeriod=25 Nov 03 19:50:29.029768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.029700 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm"] Nov 03 19:50:29.037042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.037015 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43673be-vkvlm"] Nov 03 19:50:29.374354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.374036 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:50:29.534659 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.534625 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume\") pod \"7a61bca2-e418-4af1-a018-3b377295f94e\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " Nov 03 19:50:29.534939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.534924 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn\" (UniqueName: \"kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn\") pod \"7a61bca2-e418-4af1-a018-3b377295f94e\" (UID: \"7a61bca2-e418-4af1-a018-3b377295f94e\") " Nov 03 19:50:29.535097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.535072 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume" pod "7a61bca2-e418-4af1-a018-3b377295f94e" (UID: "7a61bca2-e418-4af1-a018-3b377295f94e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:29.535565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.535546 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a61bca2-e418-4af1-a018-3b377295f94e-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:29.537645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.537622 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn" pod "7a61bca2-e418-4af1-a018-3b377295f94e" (UID: "7a61bca2-e418-4af1-a018-3b377295f94e"). InnerVolumeSpecName "kube-api-access-fcrzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:29.637657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.637581 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcrzn\" (UniqueName: \"kubernetes.io/projected/7a61bca2-e418-4af1-a018-3b377295f94e-kube-api-access-fcrzn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:29.785960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.785930 5233 generic.go:356] "Generic (PLEG): container finished" podID="7a61bca2-e418-4af1-a018-3b377295f94e" containerID="1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2" exitCode=0 Nov 03 19:50:29.786148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.785985 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" Nov 03 19:50:29.786148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.786003 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" event={"ID":"7a61bca2-e418-4af1-a018-3b377295f94e","Type":"ContainerDied","Data":"1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2"} Nov 03 19:50:29.786148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.786068 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc" event={"ID":"7a61bca2-e418-4af1-a018-3b377295f94e","Type":"ContainerDied","Data":"efd8df8bd59bbc432fc467b40f5640316810868162ea304b2c99842cd6621dec"} Nov 03 19:50:29.786148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.786092 5233 scope.go:122] "RemoveContainer" containerID="1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2" Nov 03 19:50:29.802130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.802104 5233 scope.go:122] "RemoveContainer" containerID="1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2" Nov 03 19:50:29.802727 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:29.802698 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2\": container with ID starting with 1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2 not found: ID does not exist" containerID="1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2" Nov 03 19:50:29.802859 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.802751 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2"} err="failed to get container status \"1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2\": rpc error: code = NotFound desc = could not find container \"1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2\": container with ID starting with 1f4cd88b141d5560063b0b00e265989bbc6adb04993aea99f43251728cb6fbf2 not found: ID does not exist" Nov 03 19:50:29.821180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.821138 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc"] Nov 03 19:50:29.826539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.826512 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10e551-646757c557-tttqc"] Nov 03 19:50:29.949648 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.940380 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f48abdb-9005-43b1-830d-27fbc0b3f661" path="/var/lib/kubelet/pods/1f48abdb-9005-43b1-830d-27fbc0b3f661/volumes" Nov 03 19:50:29.949648 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:29.941303 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a61bca2-e418-4af1-a018-3b377295f94e" path="/var/lib/kubelet/pods/7a61bca2-e418-4af1-a018-3b377295f94e/volumes" Nov 03 19:50:30.431009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.430933 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs"] Nov 03 19:50:30.431692 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431678 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" Nov 03 19:50:30.431737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431693 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" Nov 03 19:50:30.431737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431711 5233 cpu_manager.go:414] "containerMap: removing container" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" Nov 03 19:50:30.431737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431716 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" Nov 03 19:50:30.431737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431730 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6ee3b83c-1446-4a17-b3ca-a4086b670c5a" containerName="revoke-keys" Nov 03 19:50:30.431737 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431735 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee3b83c-1446-4a17-b3ca-a4086b670c5a" containerName="revoke-keys" Nov 03 19:50:30.431887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431743 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d4e86009-8112-45b6-a16a-854cedb6d0ee" containerName="revoke-keys" Nov 03 19:50:30.431887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431748 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e86009-8112-45b6-a16a-854cedb6d0ee" containerName="revoke-keys" Nov 03 19:50:30.431887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431766 5233 cpu_manager.go:414] "containerMap: removing container" podUID="7a61bca2-e418-4af1-a018-3b377295f94e" containerName="payload-processing" Nov 03 19:50:30.431887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431770 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a61bca2-e418-4af1-a018-3b377295f94e" containerName="payload-processing" Nov 03 19:50:30.431887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431784 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" Nov 03 19:50:30.431887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431789 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" Nov 03 19:50:30.432108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431917 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d199580a-2d32-4e95-bb54-efe923ce3397" containerName="main" Nov 03 19:50:30.432108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431926 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="294e8459-7193-46ec-81b9-5a1e0822eb92" containerName="main" Nov 03 19:50:30.432108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431934 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee3b83c-1446-4a17-b3ca-a4086b670c5a" containerName="revoke-keys" Nov 03 19:50:30.432108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431939 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e86009-8112-45b6-a16a-854cedb6d0ee" containerName="revoke-keys" Nov 03 19:50:30.432108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431947 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a61bca2-e418-4af1-a018-3b377295f94e" containerName="payload-processing" Nov 03 19:50:30.432108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.431953 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b328621b-8a79-4fe3-8c9e-2ef238b93a14" containerName="main" Nov 03 19:50:30.438265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.438243 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.445118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.445094 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs"] Nov 03 19:50:30.446337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.446313 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:30.446458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.446358 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:50:30.481115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.481083 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8p9\" (UniqueName: \"kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.481354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.481334 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.481441 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.481399 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.527709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.527684 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:50:30.587675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.586158 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8p9\" (UniqueName: \"kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.587675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.586429 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.587675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.586484 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.606503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.605964 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.611257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.611227 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.623042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.622339 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8p9\" (UniqueName: \"kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9\") pod \"maas-api-revoke-sub-6f993953-zwjcs\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:30.760324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:30.760295 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:31.062935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.062901 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs"] Nov 03 19:50:31.063457 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:31.063432 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91640c01_d010_4792_9a3a_cbe633f8a85b.slice/crio-0eb929ec68c7deec37ef5d26396689e4d02ec29d69aa08ca22f7dd504ecb3f65 WatchSource:0}: Error finding container 0eb929ec68c7deec37ef5d26396689e4d02ec29d69aa08ca22f7dd504ecb3f65: Status 404 returned error can't find the container with id 0eb929ec68c7deec37ef5d26396689e4d02ec29d69aa08ca22f7dd504ecb3f65 Nov 03 19:50:31.482993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.478904 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc"] Nov 03 19:50:31.506534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.506502 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc"] Nov 03 19:50:31.506761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.506658 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.519442 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.519415 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-477cd8-openshift-default-dockercfg-vk2t9\"" Nov 03 19:50:31.651030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.650964 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651106 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vtr\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651214 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651250 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651331 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651352 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651458 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651561 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.651675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.651661 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.753689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.753593 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.753689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.753655 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.753927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.753738 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.753927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.753768 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.753927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.753847 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.753955 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754153 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754229 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754262 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754286 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vtr\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754470 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754570 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754598 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.754747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.754706 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.756549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.756517 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.757181 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.757157 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.773290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.773268 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.780188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.780166 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vtr\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr\") pod \"e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.815815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.815785 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" event={"ID":"91640c01-d010-4792-9a3a-cbe633f8a85b","Type":"ContainerStarted","Data":"1cfab6d56d89477ceede86dcee79fa3e19c7979c87b6df85a0cd0b79397d53c2"} Nov 03 19:50:31.815994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.815833 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" event={"ID":"91640c01-d010-4792-9a3a-cbe633f8a85b","Type":"ContainerStarted","Data":"0eb929ec68c7deec37ef5d26396689e4d02ec29d69aa08ca22f7dd504ecb3f65"} Nov 03 19:50:31.829611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.829573 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:31.843368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:31.843306 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" podStartSLOduration=1.8432913800000001 podStartE2EDuration="1.84329138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:31.839235563 +0000 UTC m=+1398.180349050" watchObservedRunningTime="2026-11-03 19:50:31.84329138 +0000 UTC m=+1398.184404866" Nov 03 19:50:32.203697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.203664 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc"] Nov 03 19:50:32.232758 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:32.232718 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07afb03_6e92_444a_9250_11bf7ca89d0b.slice/crio-f8d4b81f5714533ce2c369be97a756d1d57adf813886a2ff7b154ad16aa6e7a7 WatchSource:0}: Error finding container f8d4b81f5714533ce2c369be97a756d1d57adf813886a2ff7b154ad16aa6e7a7: Status 404 returned error can't find the container with id f8d4b81f5714533ce2c369be97a756d1d57adf813886a2ff7b154ad16aa6e7a7 Nov 03 19:50:32.237064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.237000 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:50:32.237215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.237106 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:50:32.237215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.237149 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:50:32.845207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.844554 5233 generic.go:356] "Generic (PLEG): container finished" podID="91640c01-d010-4792-9a3a-cbe633f8a85b" containerID="1cfab6d56d89477ceede86dcee79fa3e19c7979c87b6df85a0cd0b79397d53c2" exitCode=0 Nov 03 19:50:32.845207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.844667 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" event={"ID":"91640c01-d010-4792-9a3a-cbe633f8a85b","Type":"ContainerDied","Data":"1cfab6d56d89477ceede86dcee79fa3e19c7979c87b6df85a0cd0b79397d53c2"} Nov 03 19:50:32.849854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.849824 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" event={"ID":"c07afb03-6e92-444a-9250-11bf7ca89d0b","Type":"ContainerStarted","Data":"3fa633fcb435a91997a91f201d81d2f98d5ff5c111fde52cf0ead29ba8f9ec04"} Nov 03 19:50:32.850037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:32.849878 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" event={"ID":"c07afb03-6e92-444a-9250-11bf7ca89d0b","Type":"ContainerStarted","Data":"f8d4b81f5714533ce2c369be97a756d1d57adf813886a2ff7b154ad16aa6e7a7"} Nov 03 19:50:33.843644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:33.842669 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:33.843949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:33.843674 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 19:50:33.843949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:33.843727 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 19:50:34.681687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.681662 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:34.740682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.740244 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podStartSLOduration=3.740201999 podStartE2EDuration="3.740201999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:32.898278356 +0000 UTC m=+1399.239391844" watchObservedRunningTime="2026-11-03 19:50:34.740201999 +0000 UTC m=+1401.081315485" Nov 03 19:50:34.746422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.746387 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca\") pod \"91640c01-d010-4792-9a3a-cbe633f8a85b\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " Nov 03 19:50:34.746591 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.746526 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca\") pod \"91640c01-d010-4792-9a3a-cbe633f8a85b\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " Nov 03 19:50:34.746803 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.746784 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9\" (UniqueName: \"kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9\") pod \"91640c01-d010-4792-9a3a-cbe633f8a85b\" (UID: \"91640c01-d010-4792-9a3a-cbe633f8a85b\") " Nov 03 19:50:34.747186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.747160 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca" pod "91640c01-d010-4792-9a3a-cbe633f8a85b" (UID: "91640c01-d010-4792-9a3a-cbe633f8a85b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:34.747695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.747673 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:34.751517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.751493 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9" pod "91640c01-d010-4792-9a3a-cbe633f8a85b" (UID: "91640c01-d010-4792-9a3a-cbe633f8a85b"). InnerVolumeSpecName "kube-api-access-dg8p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:34.751694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.751669 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca" pod "91640c01-d010-4792-9a3a-cbe633f8a85b" (UID: "91640c01-d010-4792-9a3a-cbe633f8a85b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:34.831581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.831543 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 19:50:34.831781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.831602 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 19:50:34.849885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.849855 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8p9\" (UniqueName: \"kubernetes.io/projected/91640c01-d010-4792-9a3a-cbe633f8a85b-kube-api-access-dg8p9\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:34.849885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:34.849882 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91640c01-d010-4792-9a3a-cbe633f8a85b-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:35.046965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:35.046934 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" Nov 03 19:50:35.047150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:35.046952 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs" event={"ID":"91640c01-d010-4792-9a3a-cbe633f8a85b","Type":"ContainerDied","Data":"0eb929ec68c7deec37ef5d26396689e4d02ec29d69aa08ca22f7dd504ecb3f65"} Nov 03 19:50:35.047150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:35.047053 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb929ec68c7deec37ef5d26396689e4d02ec29d69aa08ca22f7dd504ecb3f65" Nov 03 19:50:35.831086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:35.831053 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 19:50:35.831493 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:35.831125 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 19:50:36.127541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:36.127444 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:50:36.850991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:36.850263 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 19:50:36.850991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:36.850364 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 19:50:37.119733 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:37.119639 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" podUID="62f81783-cd80-4436-bc8e-d9870342432e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:50:37.832374 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:37.832338 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 19:50:37.832667 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:37.832637 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 19:50:38.479683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.479656 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:50:38.558407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.558376 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume\") pod \"62f81783-cd80-4436-bc8e-d9870342432e\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " Nov 03 19:50:38.558552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.558427 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz\" (UniqueName: \"kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz\") pod \"62f81783-cd80-4436-bc8e-d9870342432e\" (UID: \"62f81783-cd80-4436-bc8e-d9870342432e\") " Nov 03 19:50:38.558850 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.558831 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume" pod "62f81783-cd80-4436-bc8e-d9870342432e" (UID: "62f81783-cd80-4436-bc8e-d9870342432e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:38.559712 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.559671 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f81783-cd80-4436-bc8e-d9870342432e-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:38.561175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.561149 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz" pod "62f81783-cd80-4436-bc8e-d9870342432e" (UID: "62f81783-cd80-4436-bc8e-d9870342432e"). InnerVolumeSpecName "kube-api-access-k5qtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:38.634639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.633958 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:50:38.732025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.700948 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5qtz\" (UniqueName: \"kubernetes.io/projected/62f81783-cd80-4436-bc8e-d9870342432e-kube-api-access-k5qtz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:38.815997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.802745 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume\") pod \"d9702048-263e-47da-9fe4-51b24787a1d4\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " Nov 03 19:50:38.815997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.807060 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m\" (UniqueName: \"kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m\") pod \"d9702048-263e-47da-9fe4-51b24787a1d4\" (UID: \"d9702048-263e-47da-9fe4-51b24787a1d4\") " Nov 03 19:50:38.816267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.816217 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume" pod "d9702048-263e-47da-9fe4-51b24787a1d4" (UID: "d9702048-263e-47da-9fe4-51b24787a1d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:38.823457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.823430 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9702048-263e-47da-9fe4-51b24787a1d4-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:38.825452 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.825426 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m" pod "d9702048-263e-47da-9fe4-51b24787a1d4" (UID: "d9702048-263e-47da-9fe4-51b24787a1d4"). InnerVolumeSpecName "kube-api-access-kwj6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:38.830324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.830300 5233 patch_prober.go:29] interesting pod/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 03 19:50:38.830827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.830796 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 03 19:50:38.926103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.925945 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwj6m\" (UniqueName: \"kubernetes.io/projected/d9702048-263e-47da-9fe4-51b24787a1d4-kube-api-access-kwj6m\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:38.936421 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.936388 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67"] Nov 03 19:50:38.938402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.938328 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" containerName="payload-pre-processing" Nov 03 19:50:38.938402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.938358 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" containerName="payload-pre-processing" Nov 03 19:50:38.938402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.938382 5233 cpu_manager.go:414] "containerMap: removing container" podUID="62f81783-cd80-4436-bc8e-d9870342432e" containerName="payload-processing" Nov 03 19:50:38.938402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.938391 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f81783-cd80-4436-bc8e-d9870342432e" containerName="payload-processing" Nov 03 19:50:38.938402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.938407 5233 cpu_manager.go:414] "containerMap: removing container" podUID="91640c01-d010-4792-9a3a-cbe633f8a85b" containerName="revoke-keys" Nov 03 19:50:38.938754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.938416 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91640c01-d010-4792-9a3a-cbe633f8a85b" containerName="revoke-keys" Nov 03 19:50:38.939957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.939929 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f81783-cd80-4436-bc8e-d9870342432e" containerName="payload-processing" Nov 03 19:50:38.939957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.939958 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="91640c01-d010-4792-9a3a-cbe633f8a85b" containerName="revoke-keys" Nov 03 19:50:38.940130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.940003 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" containerName="payload-pre-processing" Nov 03 19:50:38.944608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.944584 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67"] Nov 03 19:50:38.944724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.944701 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:38.948628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.948605 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:38.948734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:38.948664 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:50:39.027829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.027790 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzsd\" (UniqueName: \"kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.028042 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.028016 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.028118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.028051 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.129767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.129727 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.129918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.129783 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.130150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.130108 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzsd\" (UniqueName: \"kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.130725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.130698 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.133742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.133717 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.142794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.142776 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzsd\" (UniqueName: \"kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.149534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.149507 5233 generic.go:356] "Generic (PLEG): container finished" podID="d9702048-263e-47da-9fe4-51b24787a1d4" containerID="bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16" exitCode=0 Nov 03 19:50:39.149660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.149643 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" Nov 03 19:50:39.150483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.149964 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" event={"ID":"d9702048-263e-47da-9fe4-51b24787a1d4","Type":"ContainerDied","Data":"bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16"} Nov 03 19:50:39.150753 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.150735 5233 scope.go:122] "RemoveContainer" containerID="bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16" Nov 03 19:50:39.150963 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.150933 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n" event={"ID":"d9702048-263e-47da-9fe4-51b24787a1d4","Type":"ContainerDied","Data":"414bf11780d7627fa459534cdbf620c71fefc7caef9f612d889b9b67b410a249"} Nov 03 19:50:39.152190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.152168 5233 generic.go:356] "Generic (PLEG): container finished" podID="62f81783-cd80-4436-bc8e-d9870342432e" containerID="9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa" exitCode=0 Nov 03 19:50:39.152614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.152390 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" Nov 03 19:50:39.152930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.152403 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" event={"ID":"62f81783-cd80-4436-bc8e-d9870342432e","Type":"ContainerDied","Data":"9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa"} Nov 03 19:50:39.153096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.152957 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k" event={"ID":"62f81783-cd80-4436-bc8e-d9870342432e","Type":"ContainerDied","Data":"5a12b8de281ff57ef275771d5e1f81171b6eb822df82e40136ad25e79e2a5a6f"} Nov 03 19:50:39.176632 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.176613 5233 scope.go:122] "RemoveContainer" containerID="bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16" Nov 03 19:50:39.176960 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:39.176936 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16\": container with ID starting with bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16 not found: ID does not exist" containerID="bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16" Nov 03 19:50:39.177051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.176983 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16"} err="failed to get container status \"bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16\": rpc error: code = NotFound desc = could not find container \"bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16\": container with ID starting with bfd7672aa4d0ba4043085bf09a80597de9a6ba978a1b34d3dd65ba3825896c16 not found: ID does not exist" Nov 03 19:50:39.177051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.177004 5233 scope.go:122] "RemoveContainer" containerID="9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa" Nov 03 19:50:39.199170 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.199147 5233 scope.go:122] "RemoveContainer" containerID="9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa" Nov 03 19:50:39.199863 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:39.199838 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa\": container with ID starting with 9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa not found: ID does not exist" containerID="9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa" Nov 03 19:50:39.200028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.199998 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa"} err="failed to get container status \"9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa\": rpc error: code = NotFound desc = could not find container \"9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa\": container with ID starting with 9e07f62ae775f29e7452c86cdd18ae62b029418624d33b4469f55701ff3b6faa not found: ID does not exist" Nov 03 19:50:39.202457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.202434 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n"] Nov 03 19:50:39.209196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.209173 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c5f6cb-64f7f688c5-jlr4n"] Nov 03 19:50:39.223055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.223029 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k"] Nov 03 19:50:39.229434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.229408 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c5f6cb-564c9dc85c-9zn9k"] Nov 03 19:50:39.262156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.262130 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:39.577849 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:39.577818 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4084c162_e8b9_4412_9eb0_12289eb1dc02.slice/crio-16a527ecc0af64ae648f1731a3fcadbc6d356ea8bc3eef84e130a973eac03a28 WatchSource:0}: Error finding container 16a527ecc0af64ae648f1731a3fcadbc6d356ea8bc3eef84e130a973eac03a28: Status 404 returned error can't find the container with id 16a527ecc0af64ae648f1731a3fcadbc6d356ea8bc3eef84e130a973eac03a28 Nov 03 19:50:39.578278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.577889 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67"] Nov 03 19:50:39.840919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.840846 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f81783-cd80-4436-bc8e-d9870342432e" path="/var/lib/kubelet/pods/62f81783-cd80-4436-bc8e-d9870342432e/volumes" Nov 03 19:50:39.841400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.841358 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9702048-263e-47da-9fe4-51b24787a1d4" path="/var/lib/kubelet/pods/d9702048-263e-47da-9fe4-51b24787a1d4/volumes" Nov 03 19:50:39.841751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.841738 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:39.842242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.842224 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:39.842899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:39.842881 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:50:40.168621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.168586 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" event={"ID":"4084c162-e8b9-4412-9eb0-12289eb1dc02","Type":"ContainerStarted","Data":"e2355e66d118814f96d00e5efe90843f13a2fb08ff4112a2f607c60d780c7fb4"} Nov 03 19:50:40.168786 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.168633 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" event={"ID":"4084c162-e8b9-4412-9eb0-12289eb1dc02","Type":"ContainerStarted","Data":"16a527ecc0af64ae648f1731a3fcadbc6d356ea8bc3eef84e130a973eac03a28"} Nov 03 19:50:40.195080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.195022 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" podStartSLOduration=2.195004022 podStartE2EDuration="2.195004022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:40.189477885 +0000 UTC m=+1406.530591371" watchObservedRunningTime="2026-11-03 19:50:40.195004022 +0000 UTC m=+1406.536117508" Nov 03 19:50:40.828710 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.828675 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh"] Nov 03 19:50:40.835940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.835913 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.840059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.840036 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh"] Nov 03 19:50:40.891743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.891146 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.891743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.891244 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvsl\" (UniqueName: \"kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.891743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.891442 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.993396 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.993356 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.993557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.993450 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvsl\" (UniqueName: \"kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.993635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.993620 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.994332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.994306 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:40.996136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:40.996109 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:41.005631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:41.005603 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvsl\" (UniqueName: \"kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl\") pod \"maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:41.155203 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:41.155167 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:41.188107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:41.188045 5233 generic.go:356] "Generic (PLEG): container finished" podID="4084c162-e8b9-4412-9eb0-12289eb1dc02" containerID="e2355e66d118814f96d00e5efe90843f13a2fb08ff4112a2f607c60d780c7fb4" exitCode=0 Nov 03 19:50:41.188107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:41.188087 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" event={"ID":"4084c162-e8b9-4412-9eb0-12289eb1dc02","Type":"ContainerDied","Data":"e2355e66d118814f96d00e5efe90843f13a2fb08ff4112a2f607c60d780c7fb4"} Nov 03 19:50:41.464563 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:41.464530 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d15a4f_2d64_4480_8af5_2242e11664be.slice/crio-a6a03b4dea8a489468af7192a3bbea4aba0084ceef9e35ce5b1b9783ae6e8d2f WatchSource:0}: Error finding container a6a03b4dea8a489468af7192a3bbea4aba0084ceef9e35ce5b1b9783ae6e8d2f: Status 404 returned error can't find the container with id a6a03b4dea8a489468af7192a3bbea4aba0084ceef9e35ce5b1b9783ae6e8d2f Nov 03 19:50:41.465160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:41.465133 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh"] Nov 03 19:50:42.202447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.202324 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" event={"ID":"b4d15a4f-2d64-4480-8af5-2242e11664be","Type":"ContainerStarted","Data":"c346f57942a6b1df7ff22eecf717596177c8450a3ac6fdbc517d9788669ba836"} Nov 03 19:50:42.202447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.202395 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" event={"ID":"b4d15a4f-2d64-4480-8af5-2242e11664be","Type":"ContainerStarted","Data":"a6a03b4dea8a489468af7192a3bbea4aba0084ceef9e35ce5b1b9783ae6e8d2f"} Nov 03 19:50:42.240405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.240350 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" podStartSLOduration=2.240336554 podStartE2EDuration="2.240336554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:42.237564699 +0000 UTC m=+1408.578678186" watchObservedRunningTime="2026-11-03 19:50:42.240336554 +0000 UTC m=+1408.581450039" Nov 03 19:50:42.548890 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.548865 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:42.649577 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.649542 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca\") pod \"4084c162-e8b9-4412-9eb0-12289eb1dc02\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " Nov 03 19:50:42.650486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.650459 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca\") pod \"4084c162-e8b9-4412-9eb0-12289eb1dc02\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " Nov 03 19:50:42.651089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.651066 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd\" (UniqueName: \"kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd\") pod \"4084c162-e8b9-4412-9eb0-12289eb1dc02\" (UID: \"4084c162-e8b9-4412-9eb0-12289eb1dc02\") " Nov 03 19:50:42.651724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.650199 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca" pod "4084c162-e8b9-4412-9eb0-12289eb1dc02" (UID: "4084c162-e8b9-4412-9eb0-12289eb1dc02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:42.652843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.652795 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:42.654416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.654395 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca" pod "4084c162-e8b9-4412-9eb0-12289eb1dc02" (UID: "4084c162-e8b9-4412-9eb0-12289eb1dc02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:42.654781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.654723 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd" pod "4084c162-e8b9-4412-9eb0-12289eb1dc02" (UID: "4084c162-e8b9-4412-9eb0-12289eb1dc02"). InnerVolumeSpecName "kube-api-access-2zzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:42.780594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.780492 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4084c162-e8b9-4412-9eb0-12289eb1dc02-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:42.780594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:42.780522 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzsd\" (UniqueName: \"kubernetes.io/projected/4084c162-e8b9-4412-9eb0-12289eb1dc02-kube-api-access-2zzsd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:43.423527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.423465 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" event={"ID":"4084c162-e8b9-4412-9eb0-12289eb1dc02","Type":"ContainerDied","Data":"16a527ecc0af64ae648f1731a3fcadbc6d356ea8bc3eef84e130a973eac03a28"} Nov 03 19:50:43.427992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.424113 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67" Nov 03 19:50:43.427992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.424114 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16a527ecc0af64ae648f1731a3fcadbc6d356ea8bc3eef84e130a973eac03a28" Nov 03 19:50:43.436013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.434172 5233 generic.go:356] "Generic (PLEG): container finished" podID="b4d15a4f-2d64-4480-8af5-2242e11664be" containerID="c346f57942a6b1df7ff22eecf717596177c8450a3ac6fdbc517d9788669ba836" exitCode=0 Nov 03 19:50:43.436013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.434229 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" event={"ID":"b4d15a4f-2d64-4480-8af5-2242e11664be","Type":"ContainerDied","Data":"c346f57942a6b1df7ff22eecf717596177c8450a3ac6fdbc517d9788669ba836"} Nov 03 19:50:43.506315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.506260 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj"] Nov 03 19:50:43.513309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.513282 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c5f6cb-vv4dj"] Nov 03 19:50:43.843051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:43.843017 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45238c00-498b-4fb4-8f41-62abd7bde4c9" path="/var/lib/kubelet/pods/45238c00-498b-4fb4-8f41-62abd7bde4c9/volumes" Nov 03 19:50:44.863787 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:44.863761 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:45.061756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.061725 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca\") pod \"b4d15a4f-2d64-4480-8af5-2242e11664be\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " Nov 03 19:50:45.061959 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.061894 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl\" (UniqueName: \"kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl\") pod \"b4d15a4f-2d64-4480-8af5-2242e11664be\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " Nov 03 19:50:45.062086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.062068 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca\") pod \"b4d15a4f-2d64-4480-8af5-2242e11664be\" (UID: \"b4d15a4f-2d64-4480-8af5-2242e11664be\") " Nov 03 19:50:45.062145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.062096 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca" pod "b4d15a4f-2d64-4480-8af5-2242e11664be" (UID: "b4d15a4f-2d64-4480-8af5-2242e11664be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:45.062652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.062627 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.064079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.064053 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca" pod "b4d15a4f-2d64-4480-8af5-2242e11664be" (UID: "b4d15a4f-2d64-4480-8af5-2242e11664be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:45.064161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.064077 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl" pod "b4d15a4f-2d64-4480-8af5-2242e11664be" (UID: "b4d15a4f-2d64-4480-8af5-2242e11664be"). InnerVolumeSpecName "kube-api-access-jlvsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:45.164016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.163982 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d15a4f-2d64-4480-8af5-2242e11664be-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.164016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.164012 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvsl\" (UniqueName: \"kubernetes.io/projected/b4d15a4f-2d64-4480-8af5-2242e11664be-kube-api-access-jlvsl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.518496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.516099 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" event={"ID":"b4d15a4f-2d64-4480-8af5-2242e11664be","Type":"ContainerDied","Data":"a6a03b4dea8a489468af7192a3bbea4aba0084ceef9e35ce5b1b9783ae6e8d2f"} Nov 03 19:50:45.518496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.516162 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a03b4dea8a489468af7192a3bbea4aba0084ceef9e35ce5b1b9783ae6e8d2f" Nov 03 19:50:45.518496 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:45.516232 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh" Nov 03 19:50:51.770040 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:50:51.760541 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:50:51.989475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.989444 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz"] Nov 03 19:50:51.990866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.990825 5233 cpu_manager.go:414] "containerMap: removing container" podUID="4084c162-e8b9-4412-9eb0-12289eb1dc02" containerName="revoke-keys" Nov 03 19:50:51.991066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.991050 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4084c162-e8b9-4412-9eb0-12289eb1dc02" containerName="revoke-keys" Nov 03 19:50:51.991188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.991177 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b4d15a4f-2d64-4480-8af5-2242e11664be" containerName="revoke-keys" Nov 03 19:50:51.991264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.991253 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d15a4f-2d64-4480-8af5-2242e11664be" containerName="revoke-keys" Nov 03 19:50:51.991689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.991670 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="4084c162-e8b9-4412-9eb0-12289eb1dc02" containerName="revoke-keys" Nov 03 19:50:51.991799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:51.991788 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d15a4f-2d64-4480-8af5-2242e11664be" containerName="revoke-keys" Nov 03 19:50:52.003770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.003732 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.046938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.044267 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c6a08e-dockercfg-8l9pm\"" Nov 03 19:50:52.046938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.044448 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c6a08e\"" Nov 03 19:50:52.046938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.045026 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz"] Nov 03 19:50:52.055964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.055939 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7"] Nov 03 19:50:52.218090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.169811 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkgx\" (UniqueName: \"kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx\") pod \"payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.218090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.169879 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.290014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.277363 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkgx\" (UniqueName: \"kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx\") pod \"payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.290014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.277441 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.290014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.278660 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.342770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.342691 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7"] Nov 03 19:50:52.358548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.354453 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.550993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.550439 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.551192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.551021 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghfzs\" (UniqueName: \"kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs\") pod \"payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.667123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.654928 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghfzs\" (UniqueName: \"kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs\") pod \"payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.667123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.655027 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.667123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.656765 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkgx\" (UniqueName: \"kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx\") pod \"payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.691994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.676195 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.714992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.699639 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghfzs\" (UniqueName: \"kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs\") pod \"payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:52.822512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.821301 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:52.993000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:52.987567 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:53.823776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:53.806228 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 19:50:53.823776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:53.806823 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:50:54.155470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:54.154378 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb"] Nov 03 19:50:54.155470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:54.154613 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" containerID="cri-o://a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13" gracePeriod=30 Nov 03 19:50:54.170498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:54.170463 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn"] Nov 03 19:50:54.170730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:54.170703 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" containerID="cri-o://b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6" gracePeriod=30 Nov 03 19:50:54.478048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:54.473469 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8"] Nov 03 19:50:54.478048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:54.473743 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" containerID="cri-o://0275cb42d94646bd2fb47fac741975c9159f5296410e2d99f5a836d9c03a5830" gracePeriod=30 Nov 03 19:50:54.918071 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:54.918037 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c010d8b_f1a3_4c1b_83e5_119fc481188c.slice/crio-e84eb5ae1c9241b0bc4294fe8d606fc2a94180685ad317d7268aab573b26d5a4 WatchSource:0}: Error finding container e84eb5ae1c9241b0bc4294fe8d606fc2a94180685ad317d7268aab573b26d5a4: Status 404 returned error can't find the container with id e84eb5ae1c9241b0bc4294fe8d606fc2a94180685ad317d7268aab573b26d5a4 Nov 03 19:50:55.130956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:55.130867 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz"] Nov 03 19:50:55.827995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:55.822945 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 19:50:55.976181 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:55.929724 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:56.453993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.448021 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7"] Nov 03 19:50:56.472327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.472288 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" event={"ID":"3c010d8b-f1a3-4c1b-83e5-119fc481188c","Type":"ContainerStarted","Data":"e84eb5ae1c9241b0bc4294fe8d606fc2a94180685ad317d7268aab573b26d5a4"} Nov 03 19:50:56.526019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.525652 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" event={"ID":"16792b6b-0663-4689-a3fa-16e7418eed74","Type":"ContainerStarted","Data":"d8a36666627629c1ac80f157a06d61618d7cb3cd9555c4f04d4b3f2b987ddf12"} Nov 03 19:50:56.526019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.525732 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8"] Nov 03 19:50:56.526019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.525951 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" containerID="cri-o://57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f" gracePeriod=30 Nov 03 19:50:56.772091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.772043 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:56.826865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.823246 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:56.826865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.823650 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:50:56.826865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.823716 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4" podUID="91e4a136-8e90-43e8-86c5-0b666a31cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:56.894354 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:56.894311 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:57.289012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.273425 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 03 19:50:57.289012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.285145 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:50:57.885307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.879157 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:57.885307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.880033 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:50:57.906112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.906056 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" event={"ID":"16792b6b-0663-4689-a3fa-16e7418eed74","Type":"ContainerStarted","Data":"9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d"} Nov 03 19:50:57.912015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.908333 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:57.912015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.908566 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:57.973388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.969869 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:50:57.973388 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:57.969939 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:50:58.008044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.008016 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:50:58.067931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.067907 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:50:58.068192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.068171 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" event={"ID":"3c010d8b-f1a3-4c1b-83e5-119fc481188c","Type":"ContainerStarted","Data":"948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917"} Nov 03 19:50:58.069718 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.069696 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:50:58.092716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.091034 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:58.092716 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.092310 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:50:58.138412 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.137751 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:50:58.168185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.168145 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth"] Nov 03 19:50:58.251197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.251136 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" podStartSLOduration=7.251116359 podStartE2EDuration="7.251116359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:58.041801266 +0000 UTC m=+1424.382914751" watchObservedRunningTime="2026-11-03 19:50:58.251116359 +0000 UTC m=+1424.592229849" Nov 03 19:50:58.257880 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.257852 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth"] Nov 03 19:50:58.258893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.257954 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.258893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.258736 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" podStartSLOduration=7.258721151 podStartE2EDuration="7.258721151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:58.091843783 +0000 UTC m=+1424.432957270" watchObservedRunningTime="2026-11-03 19:50:58.258721151 +0000 UTC m=+1424.599834637" Nov 03 19:50:58.262682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.262661 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:58.262818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.262794 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:50:58.414046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.413589 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.414046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.413649 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qqh\" (UniqueName: \"kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.414046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.413915 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.430579 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.430548 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9"] Nov 03 19:50:58.438237 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.438211 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10e551-dtfs9"] Nov 03 19:50:58.515895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.515858 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.516182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.516061 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.516182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.516099 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qqh\" (UniqueName: \"kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.518272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.518241 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.564994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.546684 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qqh\" (UniqueName: \"kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.576821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.572212 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-300831-b2rth\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:58.600011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:58.598236 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:50:59.235614 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:50:59.235583 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c869f2_1c4c_4e7a_9fd7_0746270237aa.slice/crio-47665f3004ff87970153c600169e6c492a060e05b2278534e958d55901bda022 WatchSource:0}: Error finding container 47665f3004ff87970153c600169e6c492a060e05b2278534e958d55901bda022: Status 404 returned error can't find the container with id 47665f3004ff87970153c600169e6c492a060e05b2278534e958d55901bda022 Nov 03 19:50:59.237133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.237103 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth"] Nov 03 19:50:59.499024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.498994 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:50:59.553618 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.553579 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.553797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.553641 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.553797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.553781 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.553994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.553959 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.554091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.554073 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.554166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.554154 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.554226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.554193 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.554294 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.554279 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.554353 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.554317 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo\") pod \"f021696d-29c5-4bcb-9996-b5227f770013\" (UID: \"f021696d-29c5-4bcb-9996-b5227f770013\") " Nov 03 19:50:59.557149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.557081 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:59.557149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.557102 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:59.557366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.557252 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:59.558225 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.558198 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:59.559779 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.559754 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:59.562363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.562342 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:50:59.565821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.565787 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "kube-api-access-lk4rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:59.567474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.567451 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:59.567920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.567883 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token" pod "f021696d-29c5-4bcb-9996-b5227f770013" (UID: "f021696d-29c5-4bcb-9996-b5227f770013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657153 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657181 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657191 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657199 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021696d-29c5-4bcb-9996-b5227f770013-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657208 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4rn\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-kube-api-access-lk4rn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657219 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657227 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021696d-29c5-4bcb-9996-b5227f770013-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657236 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021696d-29c5-4bcb-9996-b5227f770013-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.658014 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.657245 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021696d-29c5-4bcb-9996-b5227f770013-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:59.859006 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:50:59.858957 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee55c16-a923-473a-8e62-d2b025b0baa5" path="/var/lib/kubelet/pods/bee55c16-a923-473a-8e62-d2b025b0baa5/volumes" Nov 03 19:51:00.001053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.000338 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:51:00.065926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.065888 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066015 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066131 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066168 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066202 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066446 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066271 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.066446 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066439 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066526 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066576 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066539 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.066685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066585 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066621 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert\") pod \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\" (UID: \"8f360ec9-3afe-4707-96f3-b2b2f134bca7\") " Nov 03 19:51:00.066798 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.066733 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.067464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.067436 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.067575 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.067468 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.067575 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.067483 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.067827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.067803 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:00.068513 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.068492 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.071100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.071065 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:00.071769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.071746 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "kube-api-access-lqtmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:00.071898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.071774 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:00.073234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.073206 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy" pod "8f360ec9-3afe-4707-96f3-b2b2f134bca7" (UID: "8f360ec9-3afe-4707-96f3-b2b2f134bca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.085639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.085610 5233 generic.go:356] "Generic (PLEG): container finished" podID="f021696d-29c5-4bcb-9996-b5227f770013" containerID="a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13" exitCode=0 Nov 03 19:51:00.085792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.085771 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" event={"ID":"f021696d-29c5-4bcb-9996-b5227f770013","Type":"ContainerDied","Data":"a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13"} Nov 03 19:51:00.085863 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.085831 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" event={"ID":"f021696d-29c5-4bcb-9996-b5227f770013","Type":"ContainerDied","Data":"94b90404a4541a3201b8ad35d167a4096beb09fde0ca0a151b6e38c6ef3743e5"} Nov 03 19:51:00.085921 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.085863 5233 scope.go:122] "RemoveContainer" containerID="a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13" Nov 03 19:51:00.086076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.086060 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb" Nov 03 19:51:00.118172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.118101 5233 generic.go:356] "Generic (PLEG): container finished" podID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerID="b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6" exitCode=0 Nov 03 19:51:00.118293 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.118257 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" event={"ID":"8f360ec9-3afe-4707-96f3-b2b2f134bca7","Type":"ContainerDied","Data":"b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6"} Nov 03 19:51:00.118366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.118295 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" event={"ID":"8f360ec9-3afe-4707-96f3-b2b2f134bca7","Type":"ContainerDied","Data":"4e05ab5c45109cf3353f62fdfd6536ae596b6097490d4bb1f29b1eddf30b973f"} Nov 03 19:51:00.118429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.118370 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn" Nov 03 19:51:00.126947 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.126794 5233 scope.go:122] "RemoveContainer" containerID="a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13" Nov 03 19:51:00.127843 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:00.127631 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13\": container with ID starting with a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13 not found: ID does not exist" containerID="a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13" Nov 03 19:51:00.127843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.127666 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13"} err="failed to get container status \"a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13\": rpc error: code = NotFound desc = could not find container \"a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13\": container with ID starting with a99e337366358771fb133ef7d4c25d130ded11746e0c9460a65912bc52c30a13 not found: ID does not exist" Nov 03 19:51:00.127843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.127690 5233 scope.go:122] "RemoveContainer" containerID="b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6" Nov 03 19:51:00.140222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.139374 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" event={"ID":"f5c869f2-1c4c-4e7a-9fd7-0746270237aa","Type":"ContainerStarted","Data":"43f240845abca2ab64d19340c10196b94bc7b178e3f62c684739ed498b23a063"} Nov 03 19:51:00.140222 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.139436 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" event={"ID":"f5c869f2-1c4c-4e7a-9fd7-0746270237aa","Type":"ContainerStarted","Data":"47665f3004ff87970153c600169e6c492a060e05b2278534e958d55901bda022"} Nov 03 19:51:00.145104 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.145075 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:51:00.148017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.146980 5233 generic.go:356] "Generic (PLEG): container finished" podID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerID="0275cb42d94646bd2fb47fac741975c9159f5296410e2d99f5a836d9c03a5830" exitCode=0 Nov 03 19:51:00.148017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.147878 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb"] Nov 03 19:51:00.148017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.147908 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" event={"ID":"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe","Type":"ContainerDied","Data":"0275cb42d94646bd2fb47fac741975c9159f5296410e2d99f5a836d9c03a5830"} Nov 03 19:51:00.152347 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.152288 5233 scope.go:122] "RemoveContainer" containerID="b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6" Nov 03 19:51:00.152799 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:00.152703 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6\": container with ID starting with b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6 not found: ID does not exist" containerID="b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6" Nov 03 19:51:00.152799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.152753 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6"} err="failed to get container status \"b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6\": rpc error: code = NotFound desc = could not find container \"b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6\": container with ID starting with b01d2be397c72d8c39f5b7c39f3395f8be5b92b6e51e1765d323fececb68f7b6 not found: ID does not exist" Nov 03 19:51:00.152799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.152777 5233 scope.go:122] "RemoveContainer" containerID="0275cb42d94646bd2fb47fac741975c9159f5296410e2d99f5a836d9c03a5830" Nov 03 19:51:00.155336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.155240 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-ffwnb"] Nov 03 19:51:00.167053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.166998 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" podStartSLOduration=3.166964733 podStartE2EDuration="3.166964733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:00.163999343 +0000 UTC m=+1426.505112824" watchObservedRunningTime="2026-11-03 19:51:00.166964733 +0000 UTC m=+1426.508078218" Nov 03 19:51:00.168130 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168104 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.168226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168209 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168278 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168327 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168368 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168461 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168578 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168610 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.168677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168635 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.169229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168767 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.169229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168830 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.169229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.168917 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs\") pod \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\" (UID: \"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe\") " Nov 03 19:51:00.169739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.169716 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:00.170001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.169964 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.170057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.169892 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170029 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170050 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170065 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170079 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170093 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170107 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170123 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170137 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtmt\" (UniqueName: \"kubernetes.io/projected/8f360ec9-3afe-4707-96f3-b2b2f134bca7-kube-api-access-lqtmt\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.170163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.170153 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f360ec9-3afe-4707-96f3-b2b2f134bca7-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.172332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.172305 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "kube-api-access-m9vz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:00.172654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.172635 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:00.173265 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.173242 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:00.174375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.174355 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo" pod "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" (UID: "e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:00.213055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.212352 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn"] Nov 03 19:51:00.220023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.218709 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-mvrqn"] Nov 03 19:51:00.272158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.272121 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.272158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.272157 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.272400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.272174 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.272400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.272188 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vz4\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-kube-api-access-m9vz4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:00.272400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:00.272203 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:01.180611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.180558 5233 generic.go:356] "Generic (PLEG): container finished" podID="f5c869f2-1c4c-4e7a-9fd7-0746270237aa" containerID="43f240845abca2ab64d19340c10196b94bc7b178e3f62c684739ed498b23a063" exitCode=0 Nov 03 19:51:01.180611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.180674 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" event={"ID":"f5c869f2-1c4c-4e7a-9fd7-0746270237aa","Type":"ContainerDied","Data":"43f240845abca2ab64d19340c10196b94bc7b178e3f62c684739ed498b23a063"} Nov 03 19:51:01.182617 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.182585 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" event={"ID":"e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe","Type":"ContainerDied","Data":"838c2a4d82b280689a0fbdadbd4accbc0f6cbb375bb9933bbc33cb4bf711f25d"} Nov 03 19:51:01.182723 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.182597 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8" Nov 03 19:51:01.236598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.236563 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8"] Nov 03 19:51:01.245044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.245013 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c5f6cb-openshift-default-774487478c-tqnk8"] Nov 03 19:51:01.841037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.840944 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" path="/var/lib/kubelet/pods/8f360ec9-3afe-4707-96f3-b2b2f134bca7/volumes" Nov 03 19:51:01.841605 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.841582 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" path="/var/lib/kubelet/pods/e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe/volumes" Nov 03 19:51:01.842193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.842173 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f021696d-29c5-4bcb-9996-b5227f770013" path="/var/lib/kubelet/pods/f021696d-29c5-4bcb-9996-b5227f770013/volumes" Nov 03 19:51:01.934575 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:01.934549 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:51:02.005948 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.005912 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.006205 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.006187 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.006352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.006335 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.006557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.006542 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.006713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.006698 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.007344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.007322 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.007589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.007571 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.008056 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.007805 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.008358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.006753 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:02.008358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.007122 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:02.008358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.007564 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:02.008358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.008206 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:02.010162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.010124 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:02.014136 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.014110 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket\") pod \"3a12753f-6e1b-459f-baae-40742e1b158d\" (UID: \"3a12753f-6e1b-459f-baae-40742e1b158d\") " Nov 03 19:51:02.015127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.014434 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:02.015346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.015238 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.015346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.015272 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a12753f-6e1b-459f-baae-40742e1b158d-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.015346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.015286 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.015346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.015300 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.015346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.015314 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a12753f-6e1b-459f-baae-40742e1b158d-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.015346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.015328 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.020843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.020810 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:02.021090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.021061 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:02.023146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.023120 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4" pod "3a12753f-6e1b-459f-baae-40742e1b158d" (UID: "3a12753f-6e1b-459f-baae-40742e1b158d"). InnerVolumeSpecName "kube-api-access-wbkd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:02.117002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.116908 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a12753f-6e1b-459f-baae-40742e1b158d-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.117002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.116938 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.117002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.116948 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkd4\" (UniqueName: \"kubernetes.io/projected/3a12753f-6e1b-459f-baae-40742e1b158d-kube-api-access-wbkd4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.194252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.194217 5233 generic.go:356] "Generic (PLEG): container finished" podID="3a12753f-6e1b-459f-baae-40742e1b158d" containerID="57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f" exitCode=0 Nov 03 19:51:02.194735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.194269 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" Nov 03 19:51:02.194735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.194285 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" event={"ID":"3a12753f-6e1b-459f-baae-40742e1b158d","Type":"ContainerDied","Data":"57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f"} Nov 03 19:51:02.194735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.194354 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8" event={"ID":"3a12753f-6e1b-459f-baae-40742e1b158d","Type":"ContainerDied","Data":"17f7025566144136b4292df9a2fd15ca843d1c788f75cfa4abef2587330b2b1e"} Nov 03 19:51:02.194735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.194394 5233 scope.go:122] "RemoveContainer" containerID="57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f" Nov 03 19:51:02.211201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.211182 5233 scope.go:122] "RemoveContainer" containerID="57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f" Nov 03 19:51:02.211533 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:02.211510 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f\": container with ID starting with 57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f not found: ID does not exist" containerID="57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f" Nov 03 19:51:02.211837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.211539 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f"} err="failed to get container status \"57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f\": rpc error: code = NotFound desc = could not find container \"57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f\": container with ID starting with 57ff49e5637046ad6a53fce060ca1628851bec8750275054f25397874bd2fe5f not found: ID does not exist" Nov 03 19:51:02.235358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.235056 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8"] Nov 03 19:51:02.240783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.240751 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-9fsv8"] Nov 03 19:51:02.388527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.388495 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:51:02.450074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.450026 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca\") pod \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " Nov 03 19:51:02.450282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.450268 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh\" (UniqueName: \"kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh\") pod \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " Nov 03 19:51:02.450342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.450305 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca\") pod \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\" (UID: \"f5c869f2-1c4c-4e7a-9fd7-0746270237aa\") " Nov 03 19:51:02.450342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.450322 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca" pod "f5c869f2-1c4c-4e7a-9fd7-0746270237aa" (UID: "f5c869f2-1c4c-4e7a-9fd7-0746270237aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:02.450784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.450765 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.452451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.452421 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca" pod "f5c869f2-1c4c-4e7a-9fd7-0746270237aa" (UID: "f5c869f2-1c4c-4e7a-9fd7-0746270237aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:02.452554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.452452 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh" pod "f5c869f2-1c4c-4e7a-9fd7-0746270237aa" (UID: "f5c869f2-1c4c-4e7a-9fd7-0746270237aa"). InnerVolumeSpecName "kube-api-access-s4qqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:02.552773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.552741 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:02.552773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:02.552771 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4qqh\" (UniqueName: \"kubernetes.io/projected/f5c869f2-1c4c-4e7a-9fd7-0746270237aa-kube-api-access-s4qqh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:03.205874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:03.205845 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" Nov 03 19:51:03.206342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:03.205880 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth" event={"ID":"f5c869f2-1c4c-4e7a-9fd7-0746270237aa","Type":"ContainerDied","Data":"47665f3004ff87970153c600169e6c492a060e05b2278534e958d55901bda022"} Nov 03 19:51:03.206342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:03.205933 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47665f3004ff87970153c600169e6c492a060e05b2278534e958d55901bda022" Nov 03 19:51:03.884918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:03.884891 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" path="/var/lib/kubelet/pods/3a12753f-6e1b-459f-baae-40742e1b158d/volumes" Nov 03 19:51:04.517401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:04.517311 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67"] Nov 03 19:51:04.523324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:04.523292 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-477cd8-8cx67"] Nov 03 19:51:05.839776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:05.839741 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4084c162-e8b9-4412-9eb0-12289eb1dc02" path="/var/lib/kubelet/pods/4084c162-e8b9-4412-9eb0-12289eb1dc02/volumes" Nov 03 19:51:07.508078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508048 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv"] Nov 03 19:51:07.508785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508771 5233 cpu_manager.go:414] "containerMap: removing container" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" Nov 03 19:51:07.508847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508786 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" Nov 03 19:51:07.508847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508797 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" Nov 03 19:51:07.508847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508802 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" Nov 03 19:51:07.508847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508822 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f5c869f2-1c4c-4e7a-9fd7-0746270237aa" containerName="revoke-keys" Nov 03 19:51:07.508847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508829 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c869f2-1c4c-4e7a-9fd7-0746270237aa" containerName="revoke-keys" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508852 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508856 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508863 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508867 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.508999 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a12753f-6e1b-459f-baae-40742e1b158d" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.509007 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f021696d-29c5-4bcb-9996-b5227f770013" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.509018 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c869f2-1c4c-4e7a-9fd7-0746270237aa" containerName="revoke-keys" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.509029 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a8b3d6-8943-4fa5-a8c7-ed9c27b3f2fe" containerName="istio-proxy" Nov 03 19:51:07.509424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.509034 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f360ec9-3afe-4707-96f3-b2b2f134bca7" containerName="istio-proxy" Nov 03 19:51:07.514769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.514745 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.522011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.521965 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv"] Nov 03 19:51:07.539118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.538566 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.539118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.538637 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td2ql\" (UniqueName: \"kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.539118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.538682 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.586742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.586714 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:07.586955 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.586923 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:51:07.640811 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.640772 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.640962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.640825 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td2ql\" (UniqueName: \"kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.640962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.640867 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.641574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.641550 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.643267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.643242 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.663965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.663935 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td2ql\" (UniqueName: \"kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql\") pod \"maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:07.841922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:07.841847 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:08.139055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:08.138675 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv"] Nov 03 19:51:08.139577 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:08.139537 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e82824_8766_4636_a803_bf7bcc1678b7.slice/crio-bd76e6c5b28dd39aa738c792734f47acac89f7a2f9edd85aea51a575a2461217 WatchSource:0}: Error finding container bd76e6c5b28dd39aa738c792734f47acac89f7a2f9edd85aea51a575a2461217: Status 404 returned error can't find the container with id bd76e6c5b28dd39aa738c792734f47acac89f7a2f9edd85aea51a575a2461217 Nov 03 19:51:08.265015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:08.264981 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" event={"ID":"b5e82824-8766-4636-a803-bf7bcc1678b7","Type":"ContainerStarted","Data":"bd76e6c5b28dd39aa738c792734f47acac89f7a2f9edd85aea51a575a2461217"} Nov 03 19:51:09.013665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.013636 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:51:09.208207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.208183 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm"] Nov 03 19:51:09.214885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.214862 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.217012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.216984 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb"] Nov 03 19:51:09.225868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.225847 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.246004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.245963 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm"] Nov 03 19:51:09.251143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.251115 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb"] Nov 03 19:51:09.272584 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.272558 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.272743 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.272727 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.272805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.272780 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxclq\" (UniqueName: \"kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.272920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.272898 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjm6f\" (UniqueName: \"kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.273000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.272952 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.273072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.273056 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.279187 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.279168 5233 generic.go:356] "Generic (PLEG): container finished" podID="b5e82824-8766-4636-a803-bf7bcc1678b7" containerID="d230b911e760cda5a57e3e5db2f84230b5ae0cc5fa0fa7b482b67fcb5772099f" exitCode=0 Nov 03 19:51:09.279302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.279242 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" event={"ID":"b5e82824-8766-4636-a803-bf7bcc1678b7","Type":"ContainerDied","Data":"d230b911e760cda5a57e3e5db2f84230b5ae0cc5fa0fa7b482b67fcb5772099f"} Nov 03 19:51:09.374073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374034 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.374273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374134 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxclq\" (UniqueName: \"kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.374273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374210 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjm6f\" (UniqueName: \"kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.374273 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374268 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.374461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374299 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.374461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374390 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.374895 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.374864 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.375057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.375036 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.376792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.376767 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.377072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.377050 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.386760 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.386736 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjm6f\" (UniqueName: \"kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f\") pod \"maas-api-revoke-sub-3957b994-8tlwm\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.387079 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.387063 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxclq\" (UniqueName: \"kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq\") pod \"maas-api-revoke-sub-c4175961-95dnb\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.572881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.571421 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:09.583124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.582464 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:09.952349 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:09.952311 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651471bd_0e0c_4a68_9ee3_8f0b9b678328.slice/crio-49817d6777245c74fbd7df7ccd21e72340fa34d80b1324f2b582852d08c05cfb WatchSource:0}: Error finding container 49817d6777245c74fbd7df7ccd21e72340fa34d80b1324f2b582852d08c05cfb: Status 404 returned error can't find the container with id 49817d6777245c74fbd7df7ccd21e72340fa34d80b1324f2b582852d08c05cfb Nov 03 19:51:09.953707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.953686 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm"] Nov 03 19:51:09.962171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:09.962146 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb"] Nov 03 19:51:09.967926 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:09.967897 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7803a8fd_bd62_4358_a6e1_9961631b8ccd.slice/crio-9ea76fb7410c2f5ebfeebba7067426cfc4e9b010edece6562b924d77855904da WatchSource:0}: Error finding container 9ea76fb7410c2f5ebfeebba7067426cfc4e9b010edece6562b924d77855904da: Status 404 returned error can't find the container with id 9ea76fb7410c2f5ebfeebba7067426cfc4e9b010edece6562b924d77855904da Nov 03 19:51:10.122249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.117810 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh"] Nov 03 19:51:10.130028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.129934 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a033fd14-9hjrh"] Nov 03 19:51:10.171048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.169224 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:51:10.295797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.295760 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" event={"ID":"651471bd-0e0c-4a68-9ee3-8f0b9b678328","Type":"ContainerStarted","Data":"49817d6777245c74fbd7df7ccd21e72340fa34d80b1324f2b582852d08c05cfb"} Nov 03 19:51:10.297346 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.297318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" event={"ID":"7803a8fd-bd62-4358-a6e1-9961631b8ccd","Type":"ContainerStarted","Data":"9ea76fb7410c2f5ebfeebba7067426cfc4e9b010edece6562b924d77855904da"} Nov 03 19:51:10.508327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.508294 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:10.570043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.527774 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql\" (UniqueName: \"kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql\") pod \"b5e82824-8766-4636-a803-bf7bcc1678b7\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " Nov 03 19:51:10.570043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.528012 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca\") pod \"b5e82824-8766-4636-a803-bf7bcc1678b7\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " Nov 03 19:51:10.570043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.528113 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca\") pod \"b5e82824-8766-4636-a803-bf7bcc1678b7\" (UID: \"b5e82824-8766-4636-a803-bf7bcc1678b7\") " Nov 03 19:51:10.570043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.568803 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca" pod "b5e82824-8766-4636-a803-bf7bcc1678b7" (UID: "b5e82824-8766-4636-a803-bf7bcc1678b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:10.585155 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.585105 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca" pod "b5e82824-8766-4636-a803-bf7bcc1678b7" (UID: "b5e82824-8766-4636-a803-bf7bcc1678b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:10.655997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.631260 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:10.719232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.698319 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e82824-8766-4636-a803-bf7bcc1678b7-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:10.719232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.656368 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql" pod "b5e82824-8766-4636-a803-bf7bcc1678b7" (UID: "b5e82824-8766-4636-a803-bf7bcc1678b7"). InnerVolumeSpecName "kube-api-access-td2ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:10.817310 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:10.808529 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td2ql\" (UniqueName: \"kubernetes.io/projected/b5e82824-8766-4636-a803-bf7bcc1678b7-kube-api-access-td2ql\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:11.324707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.322782 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb"] Nov 03 19:51:11.324707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.324091 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" containerID="cri-o://234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7" gracePeriod=30 Nov 03 19:51:11.341066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.330547 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" event={"ID":"b5e82824-8766-4636-a803-bf7bcc1678b7","Type":"ContainerDied","Data":"bd76e6c5b28dd39aa738c792734f47acac89f7a2f9edd85aea51a575a2461217"} Nov 03 19:51:11.341066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.330601 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd76e6c5b28dd39aa738c792734f47acac89f7a2f9edd85aea51a575a2461217" Nov 03 19:51:11.341066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.330669 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv" Nov 03 19:51:11.341066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.336926 5233 generic.go:356] "Generic (PLEG): container finished" podID="651471bd-0e0c-4a68-9ee3-8f0b9b678328" containerID="193f29c6230b6a800400d978a171a8e0e49b0b1bdd40a29c2b3b61704ff15dbb" exitCode=0 Nov 03 19:51:11.341066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.337042 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" event={"ID":"651471bd-0e0c-4a68-9ee3-8f0b9b678328","Type":"ContainerDied","Data":"193f29c6230b6a800400d978a171a8e0e49b0b1bdd40a29c2b3b61704ff15dbb"} Nov 03 19:51:11.341066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.339790 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" event={"ID":"7803a8fd-bd62-4358-a6e1-9961631b8ccd","Type":"ContainerStarted","Data":"e1552bbfec6dabcd70255bd22f15fbc03c49bf08ccec28031b8e15a8c1b128cb"} Nov 03 19:51:11.449238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.448918 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" podStartSLOduration=2.448885323 podStartE2EDuration="2.448885323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:11.400641227 +0000 UTC m=+1437.741754705" watchObservedRunningTime="2026-11-03 19:51:11.448885323 +0000 UTC m=+1437.789998809" Nov 03 19:51:11.840628 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:11.840591 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d294a6-6078-4e4f-adb0-19910aacdd22" path="/var/lib/kubelet/pods/e3d294a6-6078-4e4f-adb0-19910aacdd22/volumes" Nov 03 19:51:12.353119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.353078 5233 generic.go:356] "Generic (PLEG): container finished" podID="7803a8fd-bd62-4358-a6e1-9961631b8ccd" containerID="e1552bbfec6dabcd70255bd22f15fbc03c49bf08ccec28031b8e15a8c1b128cb" exitCode=0 Nov 03 19:51:12.353506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.353163 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" event={"ID":"7803a8fd-bd62-4358-a6e1-9961631b8ccd","Type":"ContainerDied","Data":"e1552bbfec6dabcd70255bd22f15fbc03c49bf08ccec28031b8e15a8c1b128cb"} Nov 03 19:51:12.557392 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.557364 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:12.566841 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.566807 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f\" (UniqueName: \"kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f\") pod \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " Nov 03 19:51:12.566997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.566879 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca\") pod \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " Nov 03 19:51:12.566997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.566931 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca\") pod \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\" (UID: \"651471bd-0e0c-4a68-9ee3-8f0b9b678328\") " Nov 03 19:51:12.567295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.567271 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca" pod "651471bd-0e0c-4a68-9ee3-8f0b9b678328" (UID: "651471bd-0e0c-4a68-9ee3-8f0b9b678328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:12.567670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.567644 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:12.568994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.568953 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f" pod "651471bd-0e0c-4a68-9ee3-8f0b9b678328" (UID: "651471bd-0e0c-4a68-9ee3-8f0b9b678328"). InnerVolumeSpecName "kube-api-access-cjm6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:12.569064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.569039 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca" pod "651471bd-0e0c-4a68-9ee3-8f0b9b678328" (UID: "651471bd-0e0c-4a68-9ee3-8f0b9b678328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:12.669870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.669790 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjm6f\" (UniqueName: \"kubernetes.io/projected/651471bd-0e0c-4a68-9ee3-8f0b9b678328-kube-api-access-cjm6f\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:12.669870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:12.669820 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651471bd-0e0c-4a68-9ee3-8f0b9b678328-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:13.288370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.288333 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc"] Nov 03 19:51:13.288615 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.288587 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" containerID="cri-o://3fa633fcb435a91997a91f201d81d2f98d5ff5c111fde52cf0ead29ba8f9ec04" gracePeriod=30 Nov 03 19:51:13.365503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.365471 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" Nov 03 19:51:13.365939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.365509 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm" event={"ID":"651471bd-0e0c-4a68-9ee3-8f0b9b678328","Type":"ContainerDied","Data":"49817d6777245c74fbd7df7ccd21e72340fa34d80b1324f2b582852d08c05cfb"} Nov 03 19:51:13.365939 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.365557 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49817d6777245c74fbd7df7ccd21e72340fa34d80b1324f2b582852d08c05cfb" Nov 03 19:51:13.821063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.821032 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:13.880470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.880441 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca\") pod \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " Nov 03 19:51:13.880627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.880560 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq\" (UniqueName: \"kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq\") pod \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " Nov 03 19:51:13.880672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.880640 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca\") pod \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\" (UID: \"7803a8fd-bd62-4358-a6e1-9961631b8ccd\") " Nov 03 19:51:13.881084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.881052 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca" pod "7803a8fd-bd62-4358-a6e1-9961631b8ccd" (UID: "7803a8fd-bd62-4358-a6e1-9961631b8ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:13.881329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.881311 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:13.882504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.882480 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca" pod "7803a8fd-bd62-4358-a6e1-9961631b8ccd" (UID: "7803a8fd-bd62-4358-a6e1-9961631b8ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:13.882618 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.882590 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq" pod "7803a8fd-bd62-4358-a6e1-9961631b8ccd" (UID: "7803a8fd-bd62-4358-a6e1-9961631b8ccd"). InnerVolumeSpecName "kube-api-access-nxclq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:13.936154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936123 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w"] Nov 03 19:51:13.936919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936900 5233 cpu_manager.go:414] "containerMap: removing container" podUID="651471bd-0e0c-4a68-9ee3-8f0b9b678328" containerName="revoke-keys" Nov 03 19:51:13.936919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936917 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651471bd-0e0c-4a68-9ee3-8f0b9b678328" containerName="revoke-keys" Nov 03 19:51:13.937119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936951 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b5e82824-8766-4636-a803-bf7bcc1678b7" containerName="revoke-keys" Nov 03 19:51:13.937119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936956 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e82824-8766-4636-a803-bf7bcc1678b7" containerName="revoke-keys" Nov 03 19:51:13.937119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936993 5233 cpu_manager.go:414] "containerMap: removing container" podUID="7803a8fd-bd62-4358-a6e1-9961631b8ccd" containerName="revoke-keys" Nov 03 19:51:13.937119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.936999 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7803a8fd-bd62-4358-a6e1-9961631b8ccd" containerName="revoke-keys" Nov 03 19:51:13.937119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.937103 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="7803a8fd-bd62-4358-a6e1-9961631b8ccd" containerName="revoke-keys" Nov 03 19:51:13.937119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.937116 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="651471bd-0e0c-4a68-9ee3-8f0b9b678328" containerName="revoke-keys" Nov 03 19:51:13.937407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.937127 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e82824-8766-4636-a803-bf7bcc1678b7" containerName="revoke-keys" Nov 03 19:51:13.941302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.941283 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:13.966600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.961276 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr"] Nov 03 19:51:13.966600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.961555 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" containerID="cri-o://df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5" gracePeriod=30 Nov 03 19:51:13.970024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.969996 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v"] Nov 03 19:51:13.970276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.970236 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" containerID="cri-o://8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066" gracePeriod=30 Nov 03 19:51:13.976518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.976485 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w"] Nov 03 19:51:13.982678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.982654 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7803a8fd-bd62-4358-a6e1-9961631b8ccd-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:13.982678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:13.982679 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxclq\" (UniqueName: \"kubernetes.io/projected/7803a8fd-bd62-4358-a6e1-9961631b8ccd-kube-api-access-nxclq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:14.090418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.090326 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxcs\" (UniqueName: \"kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.090418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.090394 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.090629 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.090585 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.192112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.192081 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.192348 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.192330 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxcs\" (UniqueName: \"kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.192403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.192373 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.192842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.192821 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.194833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.194803 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.204878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.204851 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxcs\" (UniqueName: \"kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.268220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.268186 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:14.390444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.390400 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" event={"ID":"7803a8fd-bd62-4358-a6e1-9961631b8ccd","Type":"ContainerDied","Data":"9ea76fb7410c2f5ebfeebba7067426cfc4e9b010edece6562b924d77855904da"} Nov 03 19:51:14.390882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.390503 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea76fb7410c2f5ebfeebba7067426cfc4e9b010edece6562b924d77855904da" Nov 03 19:51:14.390882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.390604 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb" Nov 03 19:51:14.565100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:14.565065 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w"] Nov 03 19:51:15.401638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:15.401601 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" event={"ID":"7a0b1407-f67e-4433-bda0-a78e389d1cc3","Type":"ContainerStarted","Data":"660dbaa854240afc9865c54f07ba9b5ee5cc6ef434666ab389ea2666c870a45d"} Nov 03 19:51:15.402126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:15.401666 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" event={"ID":"7a0b1407-f67e-4433-bda0-a78e389d1cc3","Type":"ContainerStarted","Data":"6cd8ea67a45eedb4f762e83711672e56a2c316dbd98e79f1db7bff8fa0d57573"} Nov 03 19:51:15.463487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:15.460936 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" podStartSLOduration=2.460916547 podStartE2EDuration="2.460916547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:15.45164259 +0000 UTC m=+1441.792756076" watchObservedRunningTime="2026-11-03 19:51:15.460916547 +0000 UTC m=+1441.802030032" Nov 03 19:51:16.414008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.413944 5233 generic.go:356] "Generic (PLEG): container finished" podID="7a0b1407-f67e-4433-bda0-a78e389d1cc3" containerID="660dbaa854240afc9865c54f07ba9b5ee5cc6ef434666ab389ea2666c870a45d" exitCode=0 Nov 03 19:51:16.414359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.414000 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" event={"ID":"7a0b1407-f67e-4433-bda0-a78e389d1cc3","Type":"ContainerDied","Data":"660dbaa854240afc9865c54f07ba9b5ee5cc6ef434666ab389ea2666c870a45d"} Nov 03 19:51:16.637711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.637685 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:51:16.679023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.678995 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679066 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679133 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679244 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679306 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679280 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679302 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:16.679423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679377 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679468 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679580 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:16.679661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679593 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.679661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.679628 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy\") pod \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\" (UID: \"046e4b29-a8e2-4b02-8b84-df0d3ee25143\") " Nov 03 19:51:16.681328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.680133 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:16.681328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.680564 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.681328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.680586 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.681328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.680601 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.681328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.680872 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:16.681328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.681078 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:16.683814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.683783 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:16.683908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.683863 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "kube-api-access-8r4l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:16.686557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.686533 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:16.687331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.687307 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo" pod "046e4b29-a8e2-4b02-8b84-df0d3ee25143" (UID: "046e4b29-a8e2-4b02-8b84-df0d3ee25143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:16.789590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.789540 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.789590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.789584 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.789590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.789598 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.789923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.789611 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.789923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.789626 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r4l2\" (UniqueName: \"kubernetes.io/projected/046e4b29-a8e2-4b02-8b84-df0d3ee25143-kube-api-access-8r4l2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:16.789923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:16.789641 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4b29-a8e2-4b02-8b84-df0d3ee25143-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:17.467697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.467666 5233 generic.go:356] "Generic (PLEG): container finished" podID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerID="234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7" exitCode=0 Nov 03 19:51:17.468208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.467725 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" Nov 03 19:51:17.468208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.467747 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" event={"ID":"046e4b29-a8e2-4b02-8b84-df0d3ee25143","Type":"ContainerDied","Data":"234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7"} Nov 03 19:51:17.468208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.467811 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb" event={"ID":"046e4b29-a8e2-4b02-8b84-df0d3ee25143","Type":"ContainerDied","Data":"a99bde08b16b5fbc5a5d1abbf125b90feb9291e77d60e07aa2aebadf784030d3"} Nov 03 19:51:17.468208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.467840 5233 scope.go:122] "RemoveContainer" containerID="234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7" Nov 03 19:51:17.483367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.483342 5233 scope.go:122] "RemoveContainer" containerID="234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7" Nov 03 19:51:17.483706 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:17.483679 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7\": container with ID starting with 234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7 not found: ID does not exist" containerID="234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7" Nov 03 19:51:17.483796 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.483719 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7"} err="failed to get container status \"234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7\": rpc error: code = NotFound desc = could not find container \"234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7\": container with ID starting with 234cb492a56a171e806d828dd4705a37feed297f7c8e22b7cc42da3b5be43db7 not found: ID does not exist" Nov 03 19:51:17.530039 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.520071 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb"] Nov 03 19:51:17.533250 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.533216 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-74slb"] Nov 03 19:51:17.644385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.644352 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p"] Nov 03 19:51:17.684539 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.683856 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb"] Nov 03 19:51:17.734146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.733034 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:17.739148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.738952 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs\" (UniqueName: \"kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs\") pod \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " Nov 03 19:51:17.740596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.739433 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca\") pod \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " Nov 03 19:51:17.740596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.739600 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca\") pod \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\" (UID: \"7a0b1407-f67e-4433-bda0-a78e389d1cc3\") " Nov 03 19:51:17.744149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.744023 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca" pod "7a0b1407-f67e-4433-bda0-a78e389d1cc3" (UID: "7a0b1407-f67e-4433-bda0-a78e389d1cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:17.744149 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.744098 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca" pod "7a0b1407-f67e-4433-bda0-a78e389d1cc3" (UID: "7a0b1407-f67e-4433-bda0-a78e389d1cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:17.746868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.746754 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs" pod "7a0b1407-f67e-4433-bda0-a78e389d1cc3" (UID: "7a0b1407-f67e-4433-bda0-a78e389d1cc3"). InnerVolumeSpecName "kube-api-access-bdxcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:17.843045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.842535 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" path="/var/lib/kubelet/pods/046e4b29-a8e2-4b02-8b84-df0d3ee25143/volumes" Nov 03 19:51:17.844453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.844425 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:17.844624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.844606 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxcs\" (UniqueName: \"kubernetes.io/projected/7a0b1407-f67e-4433-bda0-a78e389d1cc3-kube-api-access-bdxcs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:17.844717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:17.844705 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b1407-f67e-4433-bda0-a78e389d1cc3-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.493091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.493055 5233 generic.go:356] "Generic (PLEG): container finished" podID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerID="3fa633fcb435a91997a91f201d81d2f98d5ff5c111fde52cf0ead29ba8f9ec04" exitCode=0 Nov 03 19:51:18.493580 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.493228 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" event={"ID":"c07afb03-6e92-444a-9250-11bf7ca89d0b","Type":"ContainerDied","Data":"3fa633fcb435a91997a91f201d81d2f98d5ff5c111fde52cf0ead29ba8f9ec04"} Nov 03 19:51:18.495360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.495327 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" event={"ID":"7a0b1407-f67e-4433-bda0-a78e389d1cc3","Type":"ContainerDied","Data":"6cd8ea67a45eedb4f762e83711672e56a2c316dbd98e79f1db7bff8fa0d57573"} Nov 03 19:51:18.495616 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.495583 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd8ea67a45eedb4f762e83711672e56a2c316dbd98e79f1db7bff8fa0d57573" Nov 03 19:51:18.495741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.495680 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w" Nov 03 19:51:18.564385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.564361 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:51:18.676080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676049 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676146 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676177 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676370 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:18.676454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676441 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676540 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676523 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676571 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676709 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676825 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.676903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.676863 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket\") pod \"c07afb03-6e92-444a-9250-11bf7ca89d0b\" (UID: \"c07afb03-6e92-444a-9250-11bf7ca89d0b\") " Nov 03 19:51:18.677929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.677902 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.678367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.678337 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:18.678934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.678545 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:18.678934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.678637 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:18.678934 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.678868 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:18.680201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.679053 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:18.680201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.679157 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "kube-api-access-l6vtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:18.680201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.679349 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:18.680522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.680498 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token" pod "c07afb03-6e92-444a-9250-11bf7ca89d0b" (UID: "c07afb03-6e92-444a-9250-11bf7ca89d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:18.779417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779381 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vtr\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-kube-api-access-l6vtr\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779416 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779432 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779442 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779451 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779459 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779467 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07afb03-6e92-444a-9250-11bf7ca89d0b-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:18.779636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:18.779475 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07afb03-6e92-444a-9250-11bf7ca89d0b-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.274252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.274222 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:51:19.324338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.324305 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:51:19.391620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391589 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.391828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391668 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.391828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391743 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.391828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391804 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.392072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391850 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.392072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391910 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.392072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.391919 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.392072 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392029 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.392282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392077 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.392282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392110 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.392282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392160 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.392282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392205 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.392282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392263 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.392547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392331 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.392547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392395 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.392547 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392487 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.392732 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392711 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.392836 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392807 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.392962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.392930 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.393326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.393302 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:19.393561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.393537 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.393678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.393660 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.393739 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.393721 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data\") pod \"2146188c-cb86-422d-9e8a-2e25495d9cd1\" (UID: \"2146188c-cb86-422d-9e8a-2e25495d9cd1\") " Nov 03 19:51:19.393873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.393840 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.393950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.393894 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket\") pod \"153c41c1-4143-4168-883d-a6b1af731ce6\" (UID: \"153c41c1-4143-4168-883d-a6b1af731ce6\") " Nov 03 19:51:19.394771 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394747 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2146188c-cb86-422d-9e8a-2e25495d9cd1-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.394877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394775 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.394877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394789 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.394877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394804 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.394877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394825 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.394877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394838 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.394877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.394852 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.395237 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.395075 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.395300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.395229 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "kube-api-access-szr5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:19.395482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.395460 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.395804 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.395786 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:19.397101 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.397063 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:19.397299 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.397257 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "kube-api-access-jk5gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:19.397908 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.397883 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:19.398161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.398005 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.398428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.398403 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:19.398582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.398563 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy" pod "2146188c-cb86-422d-9e8a-2e25495d9cd1" (UID: "2146188c-cb86-422d-9e8a-2e25495d9cd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:19.398940 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.398919 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo" pod "153c41c1-4143-4168-883d-a6b1af731ce6" (UID: "153c41c1-4143-4168-883d-a6b1af731ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:19.498401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498367 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498396 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498406 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5gq\" (UniqueName: \"kubernetes.io/projected/153c41c1-4143-4168-883d-a6b1af731ce6-kube-api-access-jk5gq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498416 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498426 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498435 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c41c1-4143-4168-883d-a6b1af731ce6-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498447 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c41c1-4143-4168-883d-a6b1af731ce6-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498461 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szr5x\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-kube-api-access-szr5x\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498473 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c41c1-4143-4168-883d-a6b1af731ce6-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498485 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.498937 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.498499 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2146188c-cb86-422d-9e8a-2e25495d9cd1-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.514740 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.514670 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" Nov 03 19:51:19.514888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.514667 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc" event={"ID":"c07afb03-6e92-444a-9250-11bf7ca89d0b","Type":"ContainerDied","Data":"f8d4b81f5714533ce2c369be97a756d1d57adf813886a2ff7b154ad16aa6e7a7"} Nov 03 19:51:19.514888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.514856 5233 scope.go:122] "RemoveContainer" containerID="3fa633fcb435a91997a91f201d81d2f98d5ff5c111fde52cf0ead29ba8f9ec04" Nov 03 19:51:19.516923 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.516898 5233 generic.go:356] "Generic (PLEG): container finished" podID="153c41c1-4143-4168-883d-a6b1af731ce6" containerID="8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066" exitCode=0 Nov 03 19:51:19.517080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.517006 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" event={"ID":"153c41c1-4143-4168-883d-a6b1af731ce6","Type":"ContainerDied","Data":"8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066"} Nov 03 19:51:19.517080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.517034 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" Nov 03 19:51:19.517080 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.517055 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v" event={"ID":"153c41c1-4143-4168-883d-a6b1af731ce6","Type":"ContainerDied","Data":"124339e5b82b7cc595c52830839b7fefbef9d8ed1046dff5e8661ef5b9c54880"} Nov 03 19:51:19.518662 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.518643 5233 generic.go:356] "Generic (PLEG): container finished" podID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerID="df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5" exitCode=0 Nov 03 19:51:19.518782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.518697 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" Nov 03 19:51:19.518782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.518731 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" event={"ID":"2146188c-cb86-422d-9e8a-2e25495d9cd1","Type":"ContainerDied","Data":"df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5"} Nov 03 19:51:19.518899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.518801 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr" event={"ID":"2146188c-cb86-422d-9e8a-2e25495d9cd1","Type":"ContainerDied","Data":"0613a92f99906c9c8169b89efa84b5d304272fc99949808babe77ff6864c6795"} Nov 03 19:51:19.528438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.528417 5233 scope.go:122] "RemoveContainer" containerID="8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066" Nov 03 19:51:19.545463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.545436 5233 scope.go:122] "RemoveContainer" containerID="a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd" Nov 03 19:51:19.559085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.558932 5233 scope.go:122] "RemoveContainer" containerID="8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066" Nov 03 19:51:19.559281 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:19.559260 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066\": container with ID starting with 8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066 not found: ID does not exist" containerID="8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066" Nov 03 19:51:19.559362 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.559294 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066"} err="failed to get container status \"8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066\": rpc error: code = NotFound desc = could not find container \"8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066\": container with ID starting with 8fe9052c37a0aa7ab533c4b5f18486dbfd116d08bc6800ec073525d1e78eb066 not found: ID does not exist" Nov 03 19:51:19.559362 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.559320 5233 scope.go:122] "RemoveContainer" containerID="a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd" Nov 03 19:51:19.559613 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:19.559589 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd\": container with ID starting with a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd not found: ID does not exist" containerID="a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd" Nov 03 19:51:19.559700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.559617 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd"} err="failed to get container status \"a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd\": rpc error: code = NotFound desc = could not find container \"a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd\": container with ID starting with a6cd624f72184764341a1e1cae3478b48d297f67c50351103ae6de7ea40665bd not found: ID does not exist" Nov 03 19:51:19.559700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.559639 5233 scope.go:122] "RemoveContainer" containerID="df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5" Nov 03 19:51:19.562460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.562440 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc"] Nov 03 19:51:19.567270 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.567248 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-477cd8-openshift-default-7b4f875647-b4jhc"] Nov 03 19:51:19.571945 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.571925 5233 scope.go:122] "RemoveContainer" containerID="df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5" Nov 03 19:51:19.572317 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:19.572293 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5\": container with ID starting with df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5 not found: ID does not exist" containerID="df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5" Nov 03 19:51:19.572406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.572319 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5"} err="failed to get container status \"df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5\": rpc error: code = NotFound desc = could not find container \"df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5\": container with ID starting with df26b23eb601a83d27cc394435cc95d62d17903e3085ac608afd9baaedcee1e5 not found: ID does not exist" Nov 03 19:51:19.577234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.577213 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v"] Nov 03 19:51:19.600993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.583693 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-tnf7v"] Nov 03 19:51:19.609255 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.609223 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr"] Nov 03 19:51:19.614440 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.614403 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-rj8cr"] Nov 03 19:51:19.868385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.864867 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" path="/var/lib/kubelet/pods/153c41c1-4143-4168-883d-a6b1af731ce6/volumes" Nov 03 19:51:19.868385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.865713 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" path="/var/lib/kubelet/pods/2146188c-cb86-422d-9e8a-2e25495d9cd1/volumes" Nov 03 19:51:19.868385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:19.866426 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" path="/var/lib/kubelet/pods/c07afb03-6e92-444a-9250-11bf7ca89d0b/volumes" Nov 03 19:51:21.225600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:21.224262 5233 scope.go:122] "RemoveContainer" containerID="bfc6034ba51c5611b82d05716468cc641a235324016d00c461e673cce4bd1358" Nov 03 19:51:21.282208 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:21.281866 5233 scope.go:122] "RemoveContainer" containerID="10e57cb0e965455a4aecb00185615d671c8233bb04ce2e125f67e3f75ffecf4d" Nov 03 19:51:21.313955 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:21.313917 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:51:21.319533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:21.319508 5233 scope.go:122] "RemoveContainer" containerID="ab6b2637f3f49997a9a796d27704282534d2cab4f923d8fd67aefdfc95a71f55" Nov 03 19:51:21.888366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:21.888333 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh"] Nov 03 19:51:21.892657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:21.892635 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ef8f84-86zkh"] Nov 03 19:51:22.644870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:22.644830 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" podUID="78363f8d-d3f4-49ea-aab2-640663ac3596" containerName="payload-processing" containerID="cri-o://14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d" gracePeriod=25 Nov 03 19:51:22.685068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:22.685026 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" podUID="e354a571-d108-451d-a82e-cf000844b8e1" containerName="payload-pre-processing" containerID="cri-o://85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4" gracePeriod=25 Nov 03 19:51:23.006552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.006528 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:51:23.066747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.066721 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:51:23.105316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.105286 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz\" (UniqueName: \"kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz\") pod \"78363f8d-d3f4-49ea-aab2-640663ac3596\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " Nov 03 19:51:23.105484 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.105362 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume\") pod \"78363f8d-d3f4-49ea-aab2-640663ac3596\" (UID: \"78363f8d-d3f4-49ea-aab2-640663ac3596\") " Nov 03 19:51:23.105813 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.105788 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume" pod "78363f8d-d3f4-49ea-aab2-640663ac3596" (UID: "78363f8d-d3f4-49ea-aab2-640663ac3596"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:23.106210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.106188 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78363f8d-d3f4-49ea-aab2-640663ac3596-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:23.107726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.107701 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz" pod "78363f8d-d3f4-49ea-aab2-640663ac3596" (UID: "78363f8d-d3f4-49ea-aab2-640663ac3596"). InnerVolumeSpecName "kube-api-access-xz6jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:23.207195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.207117 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume\") pod \"e354a571-d108-451d-a82e-cf000844b8e1\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " Nov 03 19:51:23.207424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.207404 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk\" (UniqueName: \"kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk\") pod \"e354a571-d108-451d-a82e-cf000844b8e1\" (UID: \"e354a571-d108-451d-a82e-cf000844b8e1\") " Nov 03 19:51:23.207504 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.207485 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume" pod "e354a571-d108-451d-a82e-cf000844b8e1" (UID: "e354a571-d108-451d-a82e-cf000844b8e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:23.207916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.207893 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz6jz\" (UniqueName: \"kubernetes.io/projected/78363f8d-d3f4-49ea-aab2-640663ac3596-kube-api-access-xz6jz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:23.207916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.207912 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e354a571-d108-451d-a82e-cf000844b8e1-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:23.209549 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.209524 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk" pod "e354a571-d108-451d-a82e-cf000844b8e1" (UID: "e354a571-d108-451d-a82e-cf000844b8e1"). InnerVolumeSpecName "kube-api-access-ppglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:23.309128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.309090 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppglk\" (UniqueName: \"kubernetes.io/projected/e354a571-d108-451d-a82e-cf000844b8e1-kube-api-access-ppglk\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:23.601996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.601605 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" Nov 03 19:51:23.635016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.634103 5233 generic.go:356] "Generic (PLEG): container finished" podID="78363f8d-d3f4-49ea-aab2-640663ac3596" containerID="14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d" exitCode=0 Nov 03 19:51:23.635016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.634389 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" event={"ID":"78363f8d-d3f4-49ea-aab2-640663ac3596","Type":"ContainerDied","Data":"14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d"} Nov 03 19:51:23.635016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.634420 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p" event={"ID":"78363f8d-d3f4-49ea-aab2-640663ac3596","Type":"ContainerDied","Data":"65ec19932e9ada18fe423815703e5056987a7fd978b614085b03272f5e626daf"} Nov 03 19:51:23.635016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.634441 5233 scope.go:122] "RemoveContainer" containerID="14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d" Nov 03 19:51:23.661475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.661313 5233 scope.go:122] "RemoveContainer" containerID="14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d" Nov 03 19:51:23.661828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.661646 5233 generic.go:356] "Generic (PLEG): container finished" podID="e354a571-d108-451d-a82e-cf000844b8e1" containerID="85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4" exitCode=0 Nov 03 19:51:23.661828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.661728 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" event={"ID":"e354a571-d108-451d-a82e-cf000844b8e1","Type":"ContainerDied","Data":"85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4"} Nov 03 19:51:23.661828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.661775 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" event={"ID":"e354a571-d108-451d-a82e-cf000844b8e1","Type":"ContainerDied","Data":"ef26b9e8c3103ddb8ab475ed98200fa593e6fe843fb5e5435ff5c3ca67b96439"} Nov 03 19:51:23.662026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.661850 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb" Nov 03 19:51:23.671952 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:23.671923 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d\": container with ID starting with 14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d not found: ID does not exist" containerID="14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d" Nov 03 19:51:23.672185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.672156 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d"} err="failed to get container status \"14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d\": rpc error: code = NotFound desc = could not find container \"14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d\": container with ID starting with 14c313c2ec92fb0dce762c34222e982d3310a3a2bfd370ce8dfce5becd5be95d not found: ID does not exist" Nov 03 19:51:23.672290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.672279 5233 scope.go:122] "RemoveContainer" containerID="85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4" Nov 03 19:51:23.679380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.679324 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p"] Nov 03 19:51:23.686829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.686797 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff38-c99887f97-vvz5p"] Nov 03 19:51:23.704378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.702462 5233 scope.go:122] "RemoveContainer" containerID="85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4" Nov 03 19:51:23.704378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.702570 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb"] Nov 03 19:51:23.705135 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:23.705107 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4\": container with ID starting with 85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4 not found: ID does not exist" containerID="85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4" Nov 03 19:51:23.705186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.705147 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4"} err="failed to get container status \"85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4\": rpc error: code = NotFound desc = could not find container \"85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4\": container with ID starting with 85199d3cddd4c2169104e4c6fcc75f563ba1c55ee766f7f6c4ccb63b9ff98ca4 not found: ID does not exist" Nov 03 19:51:23.709586 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.709486 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff38-7d67b5478c-9ttnb"] Nov 03 19:51:23.842148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.842116 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78363f8d-d3f4-49ea-aab2-640663ac3596" path="/var/lib/kubelet/pods/78363f8d-d3f4-49ea-aab2-640663ac3596/volumes" Nov 03 19:51:23.842619 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.842604 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d15a4f-2d64-4480-8af5-2242e11664be" path="/var/lib/kubelet/pods/b4d15a4f-2d64-4480-8af5-2242e11664be/volumes" Nov 03 19:51:23.843034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:23.843013 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e354a571-d108-451d-a82e-cf000844b8e1" path="/var/lib/kubelet/pods/e354a571-d108-451d-a82e-cf000844b8e1/volumes" Nov 03 19:51:27.501835 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.500809 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w"] Nov 03 19:51:27.510070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.510001 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5444b0-9q65w"] Nov 03 19:51:27.865249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.865152 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0b1407-f67e-4433-bda0-a78e389d1cc3" path="/var/lib/kubelet/pods/7a0b1407-f67e-4433-bda0-a78e389d1cc3/volumes" Nov 03 19:51:27.865956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.865933 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv"] Nov 03 19:51:27.867400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.867380 5233 cpu_manager.go:414] "containerMap: removing container" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" Nov 03 19:51:27.867543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.867528 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" Nov 03 19:51:27.867688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.867675 5233 cpu_manager.go:414] "containerMap: removing container" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" Nov 03 19:51:27.867795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.867781 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" Nov 03 19:51:27.867905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.867892 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" Nov 03 19:51:27.868059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868045 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" Nov 03 19:51:27.868178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868164 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" Nov 03 19:51:27.868274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868262 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" Nov 03 19:51:27.868371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868360 5233 cpu_manager.go:414] "containerMap: removing container" podUID="7a0b1407-f67e-4433-bda0-a78e389d1cc3" containerName="revoke-keys" Nov 03 19:51:27.868455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868444 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0b1407-f67e-4433-bda0-a78e389d1cc3" containerName="revoke-keys" Nov 03 19:51:27.868561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868550 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e354a571-d108-451d-a82e-cf000844b8e1" containerName="payload-pre-processing" Nov 03 19:51:27.868652 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868640 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e354a571-d108-451d-a82e-cf000844b8e1" containerName="payload-pre-processing" Nov 03 19:51:27.868756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868745 5233 cpu_manager.go:414] "containerMap: removing container" podUID="78363f8d-d3f4-49ea-aab2-640663ac3596" containerName="payload-processing" Nov 03 19:51:27.868846 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.868834 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78363f8d-d3f4-49ea-aab2-640663ac3596" containerName="payload-processing" Nov 03 19:51:27.869112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869096 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="78363f8d-d3f4-49ea-aab2-640663ac3596" containerName="payload-processing" Nov 03 19:51:27.869239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869228 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" Nov 03 19:51:27.869360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869332 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="046e4b29-a8e2-4b02-8b84-df0d3ee25143" containerName="istio-proxy" Nov 03 19:51:27.869456 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869445 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" Nov 03 19:51:27.869546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869536 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07afb03-6e92-444a-9250-11bf7ca89d0b" containerName="istio-proxy" Nov 03 19:51:27.869626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869615 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2146188c-cb86-422d-9e8a-2e25495d9cd1" containerName="istio-proxy" Nov 03 19:51:27.869725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869714 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0b1407-f67e-4433-bda0-a78e389d1cc3" containerName="revoke-keys" Nov 03 19:51:27.869820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.869808 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e354a571-d108-451d-a82e-cf000844b8e1" containerName="payload-pre-processing" Nov 03 19:51:27.875610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.875589 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv"] Nov 03 19:51:27.875687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.875677 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:27.879962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.879940 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:27.881220 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.881199 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:51:27.914015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.913988 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:27.914230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.914212 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qr8\" (UniqueName: \"kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:27.914431 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:27.914418 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.015734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.015695 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.015985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.015952 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qr8\" (UniqueName: \"kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.016248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.016227 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.016744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.016715 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.019654 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.019614 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.030378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.030316 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qr8\" (UniqueName: \"kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8\") pod \"maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.199122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.199043 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:28.500312 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.500282 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv"] Nov 03 19:51:28.501656 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:28.501615 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e07716f_cfcc_4b72_b294_96b3d2da394b.slice/crio-43adc90fcecaeff46264887ef05dd4c5053e77ae054fdc289d55682311e0f715 WatchSource:0}: Error finding container 43adc90fcecaeff46264887ef05dd4c5053e77ae054fdc289d55682311e0f715: Status 404 returned error can't find the container with id 43adc90fcecaeff46264887ef05dd4c5053e77ae054fdc289d55682311e0f715 Nov 03 19:51:28.784361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:28.784281 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" event={"ID":"6e07716f-cfcc-4b72-b294-96b3d2da394b","Type":"ContainerStarted","Data":"43adc90fcecaeff46264887ef05dd4c5053e77ae054fdc289d55682311e0f715"} Nov 03 19:51:29.795069 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:29.795028 5233 generic.go:356] "Generic (PLEG): container finished" podID="6e07716f-cfcc-4b72-b294-96b3d2da394b" containerID="e820024ba6811b5f432c43975570a1a47144fb52c27f8ba6cbf782197de769ca" exitCode=0 Nov 03 19:51:29.795480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:29.795125 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" event={"ID":"6e07716f-cfcc-4b72-b294-96b3d2da394b","Type":"ContainerDied","Data":"e820024ba6811b5f432c43975570a1a47144fb52c27f8ba6cbf782197de769ca"} Nov 03 19:51:30.162420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.162338 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz"] Nov 03 19:51:30.163169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.163149 5233 cpu_manager.go:414] "containerMap: removing container" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" Nov 03 19:51:30.163169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.163169 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c41c1-4143-4168-883d-a6b1af731ce6" containerName="istio-proxy" Nov 03 19:51:30.167336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.167315 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.175631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.175601 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz"] Nov 03 19:51:30.252463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.252430 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.252707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.252645 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvvs\" (UniqueName: \"kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.252707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.252679 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.354509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.354479 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.354733 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.354711 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvvs\" (UniqueName: \"kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.354788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.354771 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.355262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.355240 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.357124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.357098 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.371734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.371714 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvvs\" (UniqueName: \"kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.484373 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.484347 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:30.617721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.612721 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb"] Nov 03 19:51:30.617721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.613059 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" containerID="cri-o://b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691" gracePeriod=30 Nov 03 19:51:30.627929 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.627815 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj"] Nov 03 19:51:30.630286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.630245 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" containerID="cri-o://08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68" gracePeriod=30 Nov 03 19:51:30.633606 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.633572 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh"] Nov 03 19:51:30.634165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.633799 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" containerID="cri-o://2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0" gracePeriod=30 Nov 03 19:51:30.891118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:30.872722 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz"] Nov 03 19:51:30.962054 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:30.962019 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a64145f_5dee_4968_9eb8_54ce62e6e407.slice/crio-fbb7ec5597c1ac76ad97150118a8bbe6091e62ed345c94780012a01656424a29 WatchSource:0}: Error finding container fbb7ec5597c1ac76ad97150118a8bbe6091e62ed345c94780012a01656424a29: Status 404 returned error can't find the container with id fbb7ec5597c1ac76ad97150118a8bbe6091e62ed345c94780012a01656424a29 Nov 03 19:51:31.207555 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.207530 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:31.300246 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.300215 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca\") pod \"6e07716f-cfcc-4b72-b294-96b3d2da394b\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " Nov 03 19:51:31.300438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.300262 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca\") pod \"6e07716f-cfcc-4b72-b294-96b3d2da394b\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " Nov 03 19:51:31.300502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.300451 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8\" (UniqueName: \"kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8\") pod \"6e07716f-cfcc-4b72-b294-96b3d2da394b\" (UID: \"6e07716f-cfcc-4b72-b294-96b3d2da394b\") " Nov 03 19:51:31.300563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.300543 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca" pod "6e07716f-cfcc-4b72-b294-96b3d2da394b" (UID: "6e07716f-cfcc-4b72-b294-96b3d2da394b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:31.301122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.301097 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:31.385838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.385795 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca" pod "6e07716f-cfcc-4b72-b294-96b3d2da394b" (UID: "6e07716f-cfcc-4b72-b294-96b3d2da394b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:31.385985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.385794 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8" pod "6e07716f-cfcc-4b72-b294-96b3d2da394b" (UID: "6e07716f-cfcc-4b72-b294-96b3d2da394b"). InnerVolumeSpecName "kube-api-access-w7qr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:31.402425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.402395 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qr8\" (UniqueName: \"kubernetes.io/projected/6e07716f-cfcc-4b72-b294-96b3d2da394b-kube-api-access-w7qr8\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:31.402521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.402426 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e07716f-cfcc-4b72-b294-96b3d2da394b-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:31.974664 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.974621 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" Nov 03 19:51:31.975498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.975464 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv" event={"ID":"6e07716f-cfcc-4b72-b294-96b3d2da394b","Type":"ContainerDied","Data":"43adc90fcecaeff46264887ef05dd4c5053e77ae054fdc289d55682311e0f715"} Nov 03 19:51:31.975710 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.975523 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43adc90fcecaeff46264887ef05dd4c5053e77ae054fdc289d55682311e0f715" Nov 03 19:51:31.978271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.978244 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" event={"ID":"0a64145f-5dee-4968-9eb8-54ce62e6e407","Type":"ContainerStarted","Data":"a1568e54dd2d325bbb8fa066d12c89e9bfe5a7a1c67c9290e6a25cf2016a639c"} Nov 03 19:51:31.978405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:31.978287 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" event={"ID":"0a64145f-5dee-4968-9eb8-54ce62e6e407","Type":"ContainerStarted","Data":"fbb7ec5597c1ac76ad97150118a8bbe6091e62ed345c94780012a01656424a29"} Nov 03 19:51:32.014059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:32.014000 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" podStartSLOduration=2.013983332 podStartE2EDuration="2.013983332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:32.011135523 +0000 UTC m=+1458.352249008" watchObservedRunningTime="2026-11-03 19:51:32.013983332 +0000 UTC m=+1458.355096809" Nov 03 19:51:32.987884 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:32.987848 5233 generic.go:356] "Generic (PLEG): container finished" podID="0a64145f-5dee-4968-9eb8-54ce62e6e407" containerID="a1568e54dd2d325bbb8fa066d12c89e9bfe5a7a1c67c9290e6a25cf2016a639c" exitCode=0 Nov 03 19:51:32.988340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:32.987937 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" event={"ID":"0a64145f-5dee-4968-9eb8-54ce62e6e407","Type":"ContainerDied","Data":"a1568e54dd2d325bbb8fa066d12c89e9bfe5a7a1c67c9290e6a25cf2016a639c"} Nov 03 19:51:33.328602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:33.328521 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:51:33.328602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:33.328586 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:51:34.328126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.328033 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth"] Nov 03 19:51:34.336817 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.336787 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-300831-b2rth"] Nov 03 19:51:34.427933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.427899 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:34.477419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.477389 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca\") pod \"0a64145f-5dee-4968-9eb8-54ce62e6e407\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " Nov 03 19:51:34.477590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.477535 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca\") pod \"0a64145f-5dee-4968-9eb8-54ce62e6e407\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " Nov 03 19:51:34.477634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.477590 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs\" (UniqueName: \"kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs\") pod \"0a64145f-5dee-4968-9eb8-54ce62e6e407\" (UID: \"0a64145f-5dee-4968-9eb8-54ce62e6e407\") " Nov 03 19:51:34.477876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.477842 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca" pod "0a64145f-5dee-4968-9eb8-54ce62e6e407" (UID: "0a64145f-5dee-4968-9eb8-54ce62e6e407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:34.478142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.478124 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:34.479582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.479550 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca" pod "0a64145f-5dee-4968-9eb8-54ce62e6e407" (UID: "0a64145f-5dee-4968-9eb8-54ce62e6e407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:34.479758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.479732 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs" pod "0a64145f-5dee-4968-9eb8-54ce62e6e407" (UID: "0a64145f-5dee-4968-9eb8-54ce62e6e407"). InnerVolumeSpecName "kube-api-access-2wvvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:34.579340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.579267 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wvvs\" (UniqueName: \"kubernetes.io/projected/0a64145f-5dee-4968-9eb8-54ce62e6e407-kube-api-access-2wvvs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:34.579340 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.579296 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a64145f-5dee-4968-9eb8-54ce62e6e407-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:34.616053 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.616012 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv"] Nov 03 19:51:34.620951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.620917 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8cff38-hhgpv"] Nov 03 19:51:34.727889 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.727851 5233 patch_prober.go:29] interesting pod/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:51:34.728097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:34.727925 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:51:35.024813 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:35.024770 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" Nov 03 19:51:35.024813 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:35.024792 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz" event={"ID":"0a64145f-5dee-4968-9eb8-54ce62e6e407","Type":"ContainerDied","Data":"fbb7ec5597c1ac76ad97150118a8bbe6091e62ed345c94780012a01656424a29"} Nov 03 19:51:35.025069 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:35.024840 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbb7ec5597c1ac76ad97150118a8bbe6091e62ed345c94780012a01656424a29" Nov 03 19:51:35.850103 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:35.850060 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e82824-8766-4636-a803-bf7bcc1678b7" path="/var/lib/kubelet/pods/b5e82824-8766-4636-a803-bf7bcc1678b7/volumes" Nov 03 19:51:35.850612 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:35.850595 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c869f2-1c4c-4e7a-9fd7-0746270237aa" path="/var/lib/kubelet/pods/f5c869f2-1c4c-4e7a-9fd7-0746270237aa/volumes" Nov 03 19:51:35.938814 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:35.938401 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:51:36.001949 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.001628 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:51:36.007926 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.007898 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:51:36.014660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.014635 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.014734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.014706 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.014768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.014735 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.014816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.014781 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.015024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.014869 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.015163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.015070 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.015330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.015290 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.015506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.015450 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.015602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.015550 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.016058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.015832 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:36.016058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.016005 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.017861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.016551 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.017861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.017447 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.017861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.017507 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy\") pod \"006e3aa8-d443-4fda-9456-9a22856dc123\" (UID: \"006e3aa8-d443-4fda-9456-9a22856dc123\") " Nov 03 19:51:36.018490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.018298 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.018490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.018320 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.018490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.018335 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.018490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.018349 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.018490 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.018362 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e3aa8-d443-4fda-9456-9a22856dc123-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.019109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.019069 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:36.019397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.019356 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:36.020520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.020376 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.020520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.020478 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc" pod "006e3aa8-d443-4fda-9456-9a22856dc123" (UID: "006e3aa8-d443-4fda-9456-9a22856dc123"). InnerVolumeSpecName "kube-api-access-kxzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:36.048141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.037858 5233 generic.go:356] "Generic (PLEG): container finished" podID="006e3aa8-d443-4fda-9456-9a22856dc123" containerID="08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68" exitCode=0 Nov 03 19:51:36.048141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.037907 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" Nov 03 19:51:36.048141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.037948 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" event={"ID":"006e3aa8-d443-4fda-9456-9a22856dc123","Type":"ContainerDied","Data":"08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68"} Nov 03 19:51:36.048141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.038043 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj" event={"ID":"006e3aa8-d443-4fda-9456-9a22856dc123","Type":"ContainerDied","Data":"4571eaa982873dfde2d78c08e2bf4d9c0b95f7080a4c461718611fe4ee805bae"} Nov 03 19:51:36.048141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.038081 5233 scope.go:122] "RemoveContainer" containerID="08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68" Nov 03 19:51:36.052260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.052228 5233 generic.go:356] "Generic (PLEG): container finished" podID="b5d17247-e25a-435e-9310-900d1fd65797" containerID="b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691" exitCode=0 Nov 03 19:51:36.052561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.052537 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" event={"ID":"b5d17247-e25a-435e-9310-900d1fd65797","Type":"ContainerDied","Data":"b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691"} Nov 03 19:51:36.052641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.052592 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" event={"ID":"b5d17247-e25a-435e-9310-900d1fd65797","Type":"ContainerDied","Data":"89a04f081f7e67b7e50dc44b06a0333c7eb9880e26ecab39e00cee7bbc82b5ef"} Nov 03 19:51:36.052708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.052677 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb" Nov 03 19:51:36.056557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.056310 5233 generic.go:356] "Generic (PLEG): container finished" podID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerID="2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0" exitCode=0 Nov 03 19:51:36.056557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.056345 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" event={"ID":"c3133c63-b11f-4d87-88ce-c94c5f1efd13","Type":"ContainerDied","Data":"2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0"} Nov 03 19:51:36.056557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.056382 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" event={"ID":"c3133c63-b11f-4d87-88ce-c94c5f1efd13","Type":"ContainerDied","Data":"c15b5cf08c1b32ab4295c7ca740768f321fa69bdcb36545b3dc55e644af9dd5d"} Nov 03 19:51:36.056557 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.056443 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh" Nov 03 19:51:36.071327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.068100 5233 scope.go:122] "RemoveContainer" containerID="08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68" Nov 03 19:51:36.085051 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:36.071401 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68\": container with ID starting with 08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68 not found: ID does not exist" containerID="08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68" Nov 03 19:51:36.085051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.071437 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68"} err="failed to get container status \"08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68\": rpc error: code = NotFound desc = could not find container \"08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68\": container with ID starting with 08207383e50d1aec01f5089a6c4c1d21e58358cfe36f812528724cd4675eaa68 not found: ID does not exist" Nov 03 19:51:36.085051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.071460 5233 scope.go:122] "RemoveContainer" containerID="b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691" Nov 03 19:51:36.093253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.089513 5233 scope.go:122] "RemoveContainer" containerID="b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691" Nov 03 19:51:36.093253 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:36.089857 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691\": container with ID starting with b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691 not found: ID does not exist" containerID="b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691" Nov 03 19:51:36.093253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.089889 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691"} err="failed to get container status \"b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691\": rpc error: code = NotFound desc = could not find container \"b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691\": container with ID starting with b071fac86f6bc449a73108b0957848a10eba8ccb75318109d2f8e2536b204691 not found: ID does not exist" Nov 03 19:51:36.093253 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.089913 5233 scope.go:122] "RemoveContainer" containerID="2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0" Nov 03 19:51:36.096942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.096896 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj"] Nov 03 19:51:36.103566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.103522 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-mrmtj"] Nov 03 19:51:36.115164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.115143 5233 scope.go:122] "RemoveContainer" containerID="2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0" Nov 03 19:51:36.115842 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:36.115603 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0\": container with ID starting with 2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0 not found: ID does not exist" containerID="2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0" Nov 03 19:51:36.115842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.115639 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0"} err="failed to get container status \"2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0\": rpc error: code = NotFound desc = could not find container \"2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0\": container with ID starting with 2c3ee32c7fe64a56511b68e13fd5e1d10cfffc5e8b6945e7b9b1731cbbf310d0 not found: ID does not exist" Nov 03 19:51:36.119324 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119304 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.119415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119338 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.119415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119376 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.119415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119406 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.119587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119484 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.119587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119541 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.119694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119594 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.119897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119876 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.120026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119916 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.119944 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120021 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.120192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120103 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120147 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120247 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120387 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120470 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.120546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120503 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket\") pod \"b5d17247-e25a-435e-9310-900d1fd65797\" (UID: \"b5d17247-e25a-435e-9310-900d1fd65797\") " Nov 03 19:51:36.120608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.120593 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs\") pod \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\" (UID: \"c3133c63-b11f-4d87-88ce-c94c5f1efd13\") " Nov 03 19:51:36.121167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.121149 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzrc\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-kube-api-access-kxzrc\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.121260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.121171 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e3aa8-d443-4fda-9456-9a22856dc123-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.121260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.121185 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e3aa8-d443-4fda-9456-9a22856dc123-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.121260 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.121198 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e3aa8-d443-4fda-9456-9a22856dc123-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.121438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.121419 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.121676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.121658 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:36.123647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.123622 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:36.125407 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.125383 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:36.127070 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.127047 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:36.128640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.128615 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "kube-api-access-f48q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:36.128849 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.128831 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.131516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.131495 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.131683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.131666 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.132564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.132542 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.132886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.132866 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.133099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.133075 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.133186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.133145 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket" pod "b5d17247-e25a-435e-9310-900d1fd65797" (UID: "b5d17247-e25a-435e-9310-900d1fd65797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.133295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.133272 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.133439 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.133421 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:36.134475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.134454 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:51:36.135523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.135499 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:36.138551 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.138529 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs" pod "c3133c63-b11f-4d87-88ce-c94c5f1efd13" (UID: "c3133c63-b11f-4d87-88ce-c94c5f1efd13"). InnerVolumeSpecName "kube-api-access-v8tvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:36.222695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222659 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222699 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222714 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222728 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d17247-e25a-435e-9310-900d1fd65797-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222742 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222755 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222768 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d17247-e25a-435e-9310-900d1fd65797-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222785 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f48q5\" (UniqueName: \"kubernetes.io/projected/b5d17247-e25a-435e-9310-900d1fd65797-kube-api-access-f48q5\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222800 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222822 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222837 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222850 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.222870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222864 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.223271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222878 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d17247-e25a-435e-9310-900d1fd65797-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.223271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222891 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.223271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222904 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.223271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222918 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tvs\" (UniqueName: \"kubernetes.io/projected/c3133c63-b11f-4d87-88ce-c94c5f1efd13-kube-api-access-v8tvs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.223271 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.222932 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3133c63-b11f-4d87-88ce-c94c5f1efd13-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:36.389387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.389343 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb"] Nov 03 19:51:36.398083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.398052 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-whrkb"] Nov 03 19:51:36.421482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.421418 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh"] Nov 03 19:51:36.428833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:36.428804 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ef8f84-openshift-default-747457895-7kfvh"] Nov 03 19:51:37.629552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.629517 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7"] Nov 03 19:51:37.630636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630607 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" Nov 03 19:51:37.630721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630639 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" Nov 03 19:51:37.630721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630656 5233 cpu_manager.go:414] "containerMap: removing container" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" Nov 03 19:51:37.630721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630664 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" Nov 03 19:51:37.630721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630708 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6e07716f-cfcc-4b72-b294-96b3d2da394b" containerName="revoke-keys" Nov 03 19:51:37.630721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630716 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e07716f-cfcc-4b72-b294-96b3d2da394b" containerName="revoke-keys" Nov 03 19:51:37.631037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630741 5233 cpu_manager.go:414] "containerMap: removing container" podUID="0a64145f-5dee-4968-9eb8-54ce62e6e407" containerName="revoke-keys" Nov 03 19:51:37.631037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630749 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a64145f-5dee-4968-9eb8-54ce62e6e407" containerName="revoke-keys" Nov 03 19:51:37.631037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630780 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" Nov 03 19:51:37.631037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.630787 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" Nov 03 19:51:37.631037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.631003 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" containerName="istio-proxy" Nov 03 19:51:37.631037 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.631023 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" containerName="istio-proxy" Nov 03 19:51:37.631345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.631041 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e07716f-cfcc-4b72-b294-96b3d2da394b" containerName="revoke-keys" Nov 03 19:51:37.631345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.631054 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a64145f-5dee-4968-9eb8-54ce62e6e407" containerName="revoke-keys" Nov 03 19:51:37.631345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.631069 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d17247-e25a-435e-9310-900d1fd65797" containerName="istio-proxy" Nov 03 19:51:37.639735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.639709 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.642778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.642557 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7"] Nov 03 19:51:37.768899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.760009 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:37.768899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.760191 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:51:37.807000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.800271 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.807000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.800399 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk999\" (UniqueName: \"kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.807000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.800440 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.908038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.896768 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006e3aa8-d443-4fda-9456-9a22856dc123" path="/var/lib/kubelet/pods/006e3aa8-d443-4fda-9456-9a22856dc123/volumes" Nov 03 19:51:37.908038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.897539 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d17247-e25a-435e-9310-900d1fd65797" path="/var/lib/kubelet/pods/b5d17247-e25a-435e-9310-900d1fd65797/volumes" Nov 03 19:51:37.908038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.898292 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3133c63-b11f-4d87-88ce-c94c5f1efd13" path="/var/lib/kubelet/pods/c3133c63-b11f-4d87-88ce-c94c5f1efd13/volumes" Nov 03 19:51:37.914993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.910855 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.914993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.911014 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk999\" (UniqueName: \"kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.914993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.911065 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.926085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.919057 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.939058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.929333 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.950842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.950799 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk999\" (UniqueName: \"kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999\") pod \"maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:37.960458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:37.960432 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:38.312253 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:38.312223 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f57fd5_cbfc_4608_84b6_0ff8f0c87270.slice/crio-c7e6a454978d3dc6ce1073cbecf0ac8e277ebdd399c0b60e595fa9f46250d9c2 WatchSource:0}: Error finding container c7e6a454978d3dc6ce1073cbecf0ac8e277ebdd399c0b60e595fa9f46250d9c2: Status 404 returned error can't find the container with id c7e6a454978d3dc6ce1073cbecf0ac8e277ebdd399c0b60e595fa9f46250d9c2 Nov 03 19:51:38.313698 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:38.313668 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7"] Nov 03 19:51:39.176868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:39.176766 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" event={"ID":"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270","Type":"ContainerStarted","Data":"a30498d1d7c0d58f8db85d151153bf4c9c773cf1158ad7a821de5cce3ac9fba6"} Nov 03 19:51:39.177516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:39.177472 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" event={"ID":"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270","Type":"ContainerStarted","Data":"c7e6a454978d3dc6ce1073cbecf0ac8e277ebdd399c0b60e595fa9f46250d9c2"} Nov 03 19:51:39.242008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:39.241931 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" podStartSLOduration=2.241910831 podStartE2EDuration="2.241910831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:39.228214124 +0000 UTC m=+1465.569327610" watchObservedRunningTime="2026-11-03 19:51:39.241910831 +0000 UTC m=+1465.583024317" Nov 03 19:51:39.993128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:39.993087 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:40.206008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:40.205693 5233 generic.go:356] "Generic (PLEG): container finished" podID="c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" containerID="a30498d1d7c0d58f8db85d151153bf4c9c773cf1158ad7a821de5cce3ac9fba6" exitCode=0 Nov 03 19:51:40.206008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:40.205902 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" event={"ID":"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270","Type":"ContainerDied","Data":"a30498d1d7c0d58f8db85d151153bf4c9c773cf1158ad7a821de5cce3ac9fba6"} Nov 03 19:51:40.375201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:40.375099 5233 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:40.800331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:40.792847 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4"] Nov 03 19:51:41.634744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.634717 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:41.720267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.717485 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca\") pod \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " Nov 03 19:51:41.720267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.718689 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca" pod "c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" (UID: "c9f57fd5-cbfc-4608-84b6-0ff8f0c87270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:41.721197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.720685 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca\") pod \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " Nov 03 19:51:41.721197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.720762 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999\" (UniqueName: \"kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999\") pod \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\" (UID: \"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270\") " Nov 03 19:51:41.721682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.721661 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:41.726997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.725288 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca" pod "c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" (UID: "c9f57fd5-cbfc-4608-84b6-0ff8f0c87270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:41.726997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.726349 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999" pod "c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" (UID: "c9f57fd5-cbfc-4608-84b6-0ff8f0c87270"). InnerVolumeSpecName "kube-api-access-qk999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:41.823034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.822998 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:41.823034 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:41.823027 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk999\" (UniqueName: \"kubernetes.io/projected/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270-kube-api-access-qk999\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:42.237126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:42.237094 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" Nov 03 19:51:42.237307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:42.237119 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7" event={"ID":"c9f57fd5-cbfc-4608-84b6-0ff8f0c87270","Type":"ContainerDied","Data":"c7e6a454978d3dc6ce1073cbecf0ac8e277ebdd399c0b60e595fa9f46250d9c2"} Nov 03 19:51:42.237307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:42.237186 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e6a454978d3dc6ce1073cbecf0ac8e277ebdd399c0b60e595fa9f46250d9c2" Nov 03 19:51:42.293357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:42.293325 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv"] Nov 03 19:51:44.145999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.145066 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64"] Nov 03 19:51:44.146488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.146220 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" containerName="revoke-keys" Nov 03 19:51:44.146488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.146241 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" containerName="revoke-keys" Nov 03 19:51:44.146488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.146430 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" containerName="revoke-keys" Nov 03 19:51:44.158997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.156306 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.179691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.178256 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:44.179691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.178393 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:51:44.213992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.208044 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64"] Nov 03 19:51:44.339924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.339891 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hz8\" (UniqueName: \"kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.340156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.340080 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.340237 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.340222 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.441363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.441268 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.441533 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.441399 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.441615 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.441595 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hz8\" (UniqueName: \"kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.442049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.442028 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.443865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.443837 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.457782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.457757 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hz8\" (UniqueName: \"kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8\") pod \"maas-api-revoke-keys-e2e-api-b-ab584a-rdc64\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.513592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.513553 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk"] Nov 03 19:51:44.518400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.518380 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.527855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.527828 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk"] Nov 03 19:51:44.538122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.537618 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:44.644799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.644489 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.644799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.644625 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckqx\" (UniqueName: \"kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.644799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.644647 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.750624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.749053 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckqx\" (UniqueName: \"kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.750624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.749113 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.750624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.749360 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.752950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.752922 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.755434 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.755403 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.765946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.765914 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckqx\" (UniqueName: \"kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:44.832491 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.832463 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64"] Nov 03 19:51:44.833057 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:44.833031 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a41fcc_0605_486e_b1d5_3315d844c0ce.slice/crio-aa5a84e363a088848bd1a58f4552a1ac1c4142609256606dc9c887fe6bf440f0 WatchSource:0}: Error finding container aa5a84e363a088848bd1a58f4552a1ac1c4142609256606dc9c887fe6bf440f0: Status 404 returned error can't find the container with id aa5a84e363a088848bd1a58f4552a1ac1c4142609256606dc9c887fe6bf440f0 Nov 03 19:51:44.843914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:44.843894 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:45.134486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:45.134461 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk"] Nov 03 19:51:45.134733 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:45.134712 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaf7870_eca1_4b49_b3b1_ceffe8a381f4.slice/crio-9a125e9c429fc5f2fcf27fd9ea4c191e2afeb6039a2ce74376d9784e0f97dc27 WatchSource:0}: Error finding container 9a125e9c429fc5f2fcf27fd9ea4c191e2afeb6039a2ce74376d9784e0f97dc27: Status 404 returned error can't find the container with id 9a125e9c429fc5f2fcf27fd9ea4c191e2afeb6039a2ce74376d9784e0f97dc27 Nov 03 19:51:45.296646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:45.296611 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" event={"ID":"87a41fcc-0605-486e-b1d5-3315d844c0ce","Type":"ContainerStarted","Data":"53f85b79ce0f70219230e22104f1f8906a45f36f78ff84d573409c56cd4de34a"} Nov 03 19:51:45.297127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:45.296664 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" event={"ID":"87a41fcc-0605-486e-b1d5-3315d844c0ce","Type":"ContainerStarted","Data":"aa5a84e363a088848bd1a58f4552a1ac1c4142609256606dc9c887fe6bf440f0"} Nov 03 19:51:45.298318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:45.298292 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" event={"ID":"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4","Type":"ContainerStarted","Data":"9a125e9c429fc5f2fcf27fd9ea4c191e2afeb6039a2ce74376d9784e0f97dc27"} Nov 03 19:51:45.337438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:45.337380 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" podStartSLOduration=1.337362291 podStartE2EDuration="1.337362291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:45.336756755 +0000 UTC m=+1471.677870241" watchObservedRunningTime="2026-11-03 19:51:45.337362291 +0000 UTC m=+1471.678475777" Nov 03 19:51:45.801882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:45.801835 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" podUID="d1276938-c683-42a6-9ea9-35fe93da8e7a" containerName="payload-pre-processing" containerID="cri-o://987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1" gracePeriod=25 Nov 03 19:51:46.101339 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.101306 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:51:46.271501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.271399 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl\" (UniqueName: \"kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl\") pod \"d1276938-c683-42a6-9ea9-35fe93da8e7a\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " Nov 03 19:51:46.271777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.271628 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume\") pod \"d1276938-c683-42a6-9ea9-35fe93da8e7a\" (UID: \"d1276938-c683-42a6-9ea9-35fe93da8e7a\") " Nov 03 19:51:46.272089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.272050 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume" pod "d1276938-c683-42a6-9ea9-35fe93da8e7a" (UID: "d1276938-c683-42a6-9ea9-35fe93da8e7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:46.272420 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.272394 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1276938-c683-42a6-9ea9-35fe93da8e7a-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:46.274124 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.274046 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl" pod "d1276938-c683-42a6-9ea9-35fe93da8e7a" (UID: "d1276938-c683-42a6-9ea9-35fe93da8e7a"). InnerVolumeSpecName "kube-api-access-2jpwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:46.312320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.312288 5233 generic.go:356] "Generic (PLEG): container finished" podID="d1276938-c683-42a6-9ea9-35fe93da8e7a" containerID="987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1" exitCode=0 Nov 03 19:51:46.312801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.312350 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" Nov 03 19:51:46.312801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.312370 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" event={"ID":"d1276938-c683-42a6-9ea9-35fe93da8e7a","Type":"ContainerDied","Data":"987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1"} Nov 03 19:51:46.312801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.312482 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4" event={"ID":"d1276938-c683-42a6-9ea9-35fe93da8e7a","Type":"ContainerDied","Data":"56fee93b760953a393078b2d95683e61e2013a02d2cbd158f737a6a172732bfa"} Nov 03 19:51:46.312801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.312517 5233 scope.go:122] "RemoveContainer" containerID="987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1" Nov 03 19:51:46.315229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.314296 5233 generic.go:356] "Generic (PLEG): container finished" podID="bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" containerID="63d38b5cc1ed683e939614186d8b95148511c10e68705e3e8686e971bd2b5635" exitCode=0 Nov 03 19:51:46.315229 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.314596 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" event={"ID":"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4","Type":"ContainerDied","Data":"63d38b5cc1ed683e939614186d8b95148511c10e68705e3e8686e971bd2b5635"} Nov 03 19:51:46.316687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.316663 5233 generic.go:356] "Generic (PLEG): container finished" podID="87a41fcc-0605-486e-b1d5-3315d844c0ce" containerID="53f85b79ce0f70219230e22104f1f8906a45f36f78ff84d573409c56cd4de34a" exitCode=0 Nov 03 19:51:46.316813 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.316723 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" event={"ID":"87a41fcc-0605-486e-b1d5-3315d844c0ce","Type":"ContainerDied","Data":"53f85b79ce0f70219230e22104f1f8906a45f36f78ff84d573409c56cd4de34a"} Nov 03 19:51:46.326675 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.326658 5233 scope.go:122] "RemoveContainer" containerID="987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1" Nov 03 19:51:46.326960 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:46.326936 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1\": container with ID starting with 987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1 not found: ID does not exist" containerID="987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1" Nov 03 19:51:46.327052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.326984 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1"} err="failed to get container status \"987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1\": rpc error: code = NotFound desc = could not find container \"987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1\": container with ID starting with 987461c271baa5cf3def04a40c64337221486c561a1582509792a12f9f7a80a1 not found: ID does not exist" Nov 03 19:51:46.374184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.374155 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpwl\" (UniqueName: \"kubernetes.io/projected/d1276938-c683-42a6-9ea9-35fe93da8e7a-kube-api-access-2jpwl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:46.451177 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.451144 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4"] Nov 03 19:51:46.458329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.458302 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c54f85-5977cdd747-765f4"] Nov 03 19:51:46.656774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.656696 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l"] Nov 03 19:51:46.657801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.657777 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d1276938-c683-42a6-9ea9-35fe93da8e7a" containerName="payload-pre-processing" Nov 03 19:51:46.657801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.657798 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1276938-c683-42a6-9ea9-35fe93da8e7a" containerName="payload-pre-processing" Nov 03 19:51:46.657947 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.657919 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1276938-c683-42a6-9ea9-35fe93da8e7a" containerName="payload-pre-processing" Nov 03 19:51:46.662857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.662840 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.671003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.670941 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c6a08e-openshift-default-dockercfg-z5zp9\"" Nov 03 19:51:46.673083 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.673057 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l"] Nov 03 19:51:46.677986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.677948 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678102 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678047 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678168 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678275 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678210 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678342 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678325 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678501 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcxr6\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678555 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678598 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.678683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.678654 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.779831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.779798 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.779831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.779834 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.779923 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.779988 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcxr6\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780034 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780062 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780114 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780172 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780222 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780275 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780350 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780595 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.780773 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780695 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.781175 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.780846 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.782642 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.782615 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.782751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.782662 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.792740 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.792718 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.793801 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.793785 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcxr6\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6\") pod \"e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:46.980277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:46.980250 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:47.327418 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:47.327381 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f33cd4c_323e_4fe9_9820_a5e1b9342431.slice/crio-2444a5a45fd320bc6f436f9487e002c4770bc3a9f0270502e62b8b5ddb1d219f WatchSource:0}: Error finding container 2444a5a45fd320bc6f436f9487e002c4770bc3a9f0270502e62b8b5ddb1d219f: Status 404 returned error can't find the container with id 2444a5a45fd320bc6f436f9487e002c4770bc3a9f0270502e62b8b5ddb1d219f Nov 03 19:51:47.328410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.328386 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l"] Nov 03 19:51:47.332318 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.332286 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:51:47.332417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.332353 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:51:47.332417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.332387 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:51:47.609783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.609442 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:47.613668 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.613640 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:47.736391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.736360 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca\") pod \"87a41fcc-0605-486e-b1d5-3315d844c0ce\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " Nov 03 19:51:47.736708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.736688 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx\" (UniqueName: \"kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx\") pod \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " Nov 03 19:51:47.737338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.737318 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca\") pod \"87a41fcc-0605-486e-b1d5-3315d844c0ce\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " Nov 03 19:51:47.737476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.737462 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca\") pod \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " Nov 03 19:51:47.737689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.737675 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca\") pod \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\" (UID: \"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4\") " Nov 03 19:51:47.737833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.737820 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8\" (UniqueName: \"kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8\") pod \"87a41fcc-0605-486e-b1d5-3315d844c0ce\" (UID: \"87a41fcc-0605-486e-b1d5-3315d844c0ce\") " Nov 03 19:51:47.740614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.740306 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca" pod "bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" (UID: "bbaf7870-eca1-4b49-b3b1-ceffe8a381f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:47.740614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.740556 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca" pod "87a41fcc-0605-486e-b1d5-3315d844c0ce" (UID: "87a41fcc-0605-486e-b1d5-3315d844c0ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:47.777992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.768029 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca" pod "87a41fcc-0605-486e-b1d5-3315d844c0ce" (UID: "87a41fcc-0605-486e-b1d5-3315d844c0ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:47.781651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.781608 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr"] Nov 03 19:51:47.806219 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.806171 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx" pod "bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" (UID: "bbaf7870-eca1-4b49-b3b1-ceffe8a381f4"). InnerVolumeSpecName "kube-api-access-gckqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:47.807541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.807489 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca" pod "bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" (UID: "bbaf7870-eca1-4b49-b3b1-ceffe8a381f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:47.813170 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.810128 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8" pod "87a41fcc-0605-486e-b1d5-3315d844c0ce" (UID: "87a41fcc-0605-486e-b1d5-3315d844c0ce"). InnerVolumeSpecName "kube-api-access-85hz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:47.840279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.840202 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.840279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.840239 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.840279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.840257 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.840279 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.840272 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hz8\" (UniqueName: \"kubernetes.io/projected/87a41fcc-0605-486e-b1d5-3315d844c0ce-kube-api-access-85hz8\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.840626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.840288 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a41fcc-0605-486e-b1d5-3315d844c0ce-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.840626 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.840301 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gckqx\" (UniqueName: \"kubernetes.io/projected/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4-kube-api-access-gckqx\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.864349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:47.864304 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1276938-c683-42a6-9ea9-35fe93da8e7a" path="/var/lib/kubelet/pods/d1276938-c683-42a6-9ea9-35fe93da8e7a/volumes" Nov 03 19:51:48.342927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.342837 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" Nov 03 19:51:48.343465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.342849 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk" event={"ID":"bbaf7870-eca1-4b49-b3b1-ceffe8a381f4","Type":"ContainerDied","Data":"9a125e9c429fc5f2fcf27fd9ea4c191e2afeb6039a2ce74376d9784e0f97dc27"} Nov 03 19:51:48.343465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.343210 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a125e9c429fc5f2fcf27fd9ea4c191e2afeb6039a2ce74376d9784e0f97dc27" Nov 03 19:51:48.345292 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.345264 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" event={"ID":"5f33cd4c-323e-4fe9-9820-a5e1b9342431","Type":"ContainerStarted","Data":"bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b"} Nov 03 19:51:48.345522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.345500 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" event={"ID":"5f33cd4c-323e-4fe9-9820-a5e1b9342431","Type":"ContainerStarted","Data":"2444a5a45fd320bc6f436f9487e002c4770bc3a9f0270502e62b8b5ddb1d219f"} Nov 03 19:51:48.347088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.347067 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" Nov 03 19:51:48.347181 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.347064 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64" event={"ID":"87a41fcc-0605-486e-b1d5-3315d844c0ce","Type":"ContainerDied","Data":"aa5a84e363a088848bd1a58f4552a1ac1c4142609256606dc9c887fe6bf440f0"} Nov 03 19:51:48.347248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.347196 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5a84e363a088848bd1a58f4552a1ac1c4142609256606dc9c887fe6bf440f0" Nov 03 19:51:48.382693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.382656 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podStartSLOduration=2.38264353 podStartE2EDuration="2.38264353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:48.381265398 +0000 UTC m=+1474.722378886" watchObservedRunningTime="2026-11-03 19:51:48.38264353 +0000 UTC m=+1474.723757017" Nov 03 19:51:48.980548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.980518 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:48.982338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.982310 5233 patch_prober.go:29] interesting pod/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:51:48.982471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:48.982361 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 19:51:49.990003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:49.986430 5233 patch_prober.go:29] interesting pod/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:51:49.990003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:49.986510 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 19:51:50.244699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:50.244627 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz"] Nov 03 19:51:50.249176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:50.249147 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c54f85-2x9lz"] Nov 03 19:51:50.981526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:50.981491 5233 patch_prober.go:29] interesting pod/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:51:50.981704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:50.981575 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 19:51:51.278450 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:51.278366 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:51:51.838735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:51.838707 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a64145f-5dee-4968-9eb8-54ce62e6e407" path="/var/lib/kubelet/pods/0a64145f-5dee-4968-9eb8-54ce62e6e407/volumes" Nov 03 19:51:51.980867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:51.980827 5233 patch_prober.go:29] interesting pod/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:51:51.981104 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:51.980880 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 19:51:52.677429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.677395 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x"] Nov 03 19:51:52.678182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.678165 5233 cpu_manager.go:414] "containerMap: removing container" podUID="87a41fcc-0605-486e-b1d5-3315d844c0ce" containerName="revoke-keys" Nov 03 19:51:52.678226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.678183 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a41fcc-0605-486e-b1d5-3315d844c0ce" containerName="revoke-keys" Nov 03 19:51:52.678226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.678198 5233 cpu_manager.go:414] "containerMap: removing container" podUID="bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" containerName="revoke-keys" Nov 03 19:51:52.678226 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.678204 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" containerName="revoke-keys" Nov 03 19:51:52.678338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.678300 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a41fcc-0605-486e-b1d5-3315d844c0ce" containerName="revoke-keys" Nov 03 19:51:52.678338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.678311 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" containerName="revoke-keys" Nov 03 19:51:52.683201 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.683184 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.719367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.714255 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:52.719367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.714565 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:51:52.733334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.733295 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx"] Nov 03 19:51:52.741717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.741690 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.746452 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.746426 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x"] Nov 03 19:51:52.750697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.750669 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhw8m\" (UniqueName: \"kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.750874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.750853 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.751041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.751024 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.753708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.753689 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx"] Nov 03 19:51:52.783235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.783190 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" podUID="f24de34e-fb61-4398-b65f-7d2c552a91a6" containerName="payload-processing" containerID="cri-o://2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab" gracePeriod=25 Nov 03 19:51:52.851776 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.851747 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.851896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.851795 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.851896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.851852 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhw8m\" (UniqueName: \"kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.852082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.852059 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhhzc\" (UniqueName: \"kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.852176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.852143 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.852239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.852209 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.852526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.852500 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.854594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.854575 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.864351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.864332 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhw8m\" (UniqueName: \"kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m\") pod \"maas-api-revoke-sub-5df7053b-sh86x\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:52.954108 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.954079 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhhzc\" (UniqueName: \"kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.954259 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.954174 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.954326 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.954267 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.954956 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.954932 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.956423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.956402 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.968449 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.968426 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhhzc\" (UniqueName: \"kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc\") pod \"maas-api-revoke-sub-9031b44a-lspxx\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:52.981307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.981283 5233 patch_prober.go:29] interesting pod/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:51:52.981403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:52.981344 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 19:51:53.041885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.041591 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:53.068994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.066528 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:53.269462 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.269050 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:51:53.380486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.380290 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume\") pod \"f24de34e-fb61-4398-b65f-7d2c552a91a6\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " Nov 03 19:51:53.380486 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.380365 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94\" (UniqueName: \"kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94\") pod \"f24de34e-fb61-4398-b65f-7d2c552a91a6\" (UID: \"f24de34e-fb61-4398-b65f-7d2c552a91a6\") " Nov 03 19:51:53.381352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.381321 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume" pod "f24de34e-fb61-4398-b65f-7d2c552a91a6" (UID: "f24de34e-fb61-4398-b65f-7d2c552a91a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:53.383264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.383235 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94" pod "f24de34e-fb61-4398-b65f-7d2c552a91a6" (UID: "f24de34e-fb61-4398-b65f-7d2c552a91a6"). InnerVolumeSpecName "kube-api-access-wzq94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:53.441831 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.441796 5233 generic.go:356] "Generic (PLEG): container finished" podID="f24de34e-fb61-4398-b65f-7d2c552a91a6" containerID="2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab" exitCode=0 Nov 03 19:51:53.442036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.441860 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" Nov 03 19:51:53.442036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.441879 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" event={"ID":"f24de34e-fb61-4398-b65f-7d2c552a91a6","Type":"ContainerDied","Data":"2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab"} Nov 03 19:51:53.442036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.441934 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr" event={"ID":"f24de34e-fb61-4398-b65f-7d2c552a91a6","Type":"ContainerDied","Data":"e3a8a55e64258a14c0fb5f27fa02c4dc2098efe0ca108e0f59400b0580ea1f19"} Nov 03 19:51:53.442036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.441956 5233 scope.go:122] "RemoveContainer" containerID="2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab" Nov 03 19:51:53.456431 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.456410 5233 scope.go:122] "RemoveContainer" containerID="2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab" Nov 03 19:51:53.456738 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:51:53.456720 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab\": container with ID starting with 2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab not found: ID does not exist" containerID="2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab" Nov 03 19:51:53.456793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.456749 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab"} err="failed to get container status \"2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab\": rpc error: code = NotFound desc = could not find container \"2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab\": container with ID starting with 2ea7b0afa8f66bdd702caa2f570671125dc9874a2cef1b2255f87e0458aaf5ab not found: ID does not exist" Nov 03 19:51:53.475011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.474985 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx"] Nov 03 19:51:53.476575 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:53.476551 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f58bd3_f5bc_4b96_9203_9bfb637d58ee.slice/crio-123972c841835d4dd3536a215b5e4108b7a22ad48c10d8ffbf79fcc68342c407 WatchSource:0}: Error finding container 123972c841835d4dd3536a215b5e4108b7a22ad48c10d8ffbf79fcc68342c407: Status 404 returned error can't find the container with id 123972c841835d4dd3536a215b5e4108b7a22ad48c10d8ffbf79fcc68342c407 Nov 03 19:51:53.480107 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.480085 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr"] Nov 03 19:51:53.482380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.482298 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f24de34e-fb61-4398-b65f-7d2c552a91a6-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:53.482380 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.482322 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzq94\" (UniqueName: \"kubernetes.io/projected/f24de34e-fb61-4398-b65f-7d2c552a91a6-kube-api-access-wzq94\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:53.522451 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:53.522410 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c19abf_04cd_4b8c_8edc_88fa7c0bddda.slice/crio-f3b02ccfde340505932e56d2710240a8cdea33dde876e009a1fa079a00c0c467 WatchSource:0}: Error finding container f3b02ccfde340505932e56d2710240a8cdea33dde876e009a1fa079a00c0c467: Status 404 returned error can't find the container with id f3b02ccfde340505932e56d2710240a8cdea33dde876e009a1fa079a00c0c467 Nov 03 19:51:53.525608 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.525579 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x"] Nov 03 19:51:53.530768 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.530735 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2c3fc4-57b4bff8c-8thrr"] Nov 03 19:51:53.849082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.849055 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24de34e-fb61-4398-b65f-7d2c552a91a6" path="/var/lib/kubelet/pods/f24de34e-fb61-4398-b65f-7d2c552a91a6/volumes" Nov 03 19:51:53.981315 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.981284 5233 patch_prober.go:29] interesting pod/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 19:51:53.981488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:53.981339 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 19:51:54.296964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.296925 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq"] Nov 03 19:51:54.456689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.456584 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" event={"ID":"68f58bd3-f5bc-4b96-9203-9bfb637d58ee","Type":"ContainerStarted","Data":"8339db474e81510ca148cbfeab2d56d6aa4a5528bff647d4963c08d5eed23acf"} Nov 03 19:51:54.456689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.456663 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" event={"ID":"68f58bd3-f5bc-4b96-9203-9bfb637d58ee","Type":"ContainerStarted","Data":"123972c841835d4dd3536a215b5e4108b7a22ad48c10d8ffbf79fcc68342c407"} Nov 03 19:51:54.458681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.458655 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" event={"ID":"32c19abf-04cd-4b8c-8edc-88fa7c0bddda","Type":"ContainerStarted","Data":"b4c27f8f96f718cffa0dc4b45a10c42e719687fa122be25616e2a4f7e55bf517"} Nov 03 19:51:54.458837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.458698 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" event={"ID":"32c19abf-04cd-4b8c-8edc-88fa7c0bddda","Type":"ContainerStarted","Data":"f3b02ccfde340505932e56d2710240a8cdea33dde876e009a1fa079a00c0c467"} Nov 03 19:51:54.477059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.477016 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" podStartSLOduration=2.476964726 podStartE2EDuration="2.476964726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:54.475478767 +0000 UTC m=+1480.816592252" watchObservedRunningTime="2026-11-03 19:51:54.476964726 +0000 UTC m=+1480.818078245" Nov 03 19:51:54.499487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.499123 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" podStartSLOduration=2.499105858 podStartE2EDuration="2.499105858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:54.497080564 +0000 UTC m=+1480.838194049" watchObservedRunningTime="2026-11-03 19:51:54.499105858 +0000 UTC m=+1480.840219344" Nov 03 19:51:54.665031 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.664801 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx"] Nov 03 19:51:54.985403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.985368 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:54.985823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.985801 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:54.986687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:54.986668 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:51:55.469599 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:55.469567 5233 generic.go:356] "Generic (PLEG): container finished" podID="68f58bd3-f5bc-4b96-9203-9bfb637d58ee" containerID="8339db474e81510ca148cbfeab2d56d6aa4a5528bff647d4963c08d5eed23acf" exitCode=0 Nov 03 19:51:55.469806 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:55.469650 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" event={"ID":"68f58bd3-f5bc-4b96-9203-9bfb637d58ee","Type":"ContainerDied","Data":"8339db474e81510ca148cbfeab2d56d6aa4a5528bff647d4963c08d5eed23acf"} Nov 03 19:51:55.472202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:55.472175 5233 generic.go:356] "Generic (PLEG): container finished" podID="32c19abf-04cd-4b8c-8edc-88fa7c0bddda" containerID="b4c27f8f96f718cffa0dc4b45a10c42e719687fa122be25616e2a4f7e55bf517" exitCode=0 Nov 03 19:51:55.473118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:55.473049 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" event={"ID":"32c19abf-04cd-4b8c-8edc-88fa7c0bddda","Type":"ContainerDied","Data":"b4c27f8f96f718cffa0dc4b45a10c42e719687fa122be25616e2a4f7e55bf517"} Nov 03 19:51:56.676887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.676860 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:56.745904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.745842 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:56.828515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.828471 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc\" (UniqueName: \"kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc\") pod \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " Nov 03 19:51:56.828781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.828536 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca\") pod \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " Nov 03 19:51:56.828781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.828619 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m\" (UniqueName: \"kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m\") pod \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " Nov 03 19:51:56.828781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.828713 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca\") pod \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " Nov 03 19:51:56.829036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.828834 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca\") pod \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\" (UID: \"68f58bd3-f5bc-4b96-9203-9bfb637d58ee\") " Nov 03 19:51:56.829036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.828881 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca\") pod \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\" (UID: \"32c19abf-04cd-4b8c-8edc-88fa7c0bddda\") " Nov 03 19:51:56.829530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.829506 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca" pod "68f58bd3-f5bc-4b96-9203-9bfb637d58ee" (UID: "68f58bd3-f5bc-4b96-9203-9bfb637d58ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:56.830094 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.830070 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca" pod "32c19abf-04cd-4b8c-8edc-88fa7c0bddda" (UID: "32c19abf-04cd-4b8c-8edc-88fa7c0bddda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:56.831761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.831736 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca" pod "68f58bd3-f5bc-4b96-9203-9bfb637d58ee" (UID: "68f58bd3-f5bc-4b96-9203-9bfb637d58ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:56.832126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.832106 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca" pod "32c19abf-04cd-4b8c-8edc-88fa7c0bddda" (UID: "32c19abf-04cd-4b8c-8edc-88fa7c0bddda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:56.832361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.832341 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m" pod "32c19abf-04cd-4b8c-8edc-88fa7c0bddda" (UID: "32c19abf-04cd-4b8c-8edc-88fa7c0bddda"). InnerVolumeSpecName "kube-api-access-fhw8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:56.833898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.833876 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc" pod "68f58bd3-f5bc-4b96-9203-9bfb637d58ee" (UID: "68f58bd3-f5bc-4b96-9203-9bfb637d58ee"). InnerVolumeSpecName "kube-api-access-dhhzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:56.931159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.931120 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:56.931159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.931159 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:56.931343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.931176 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhhzc\" (UniqueName: \"kubernetes.io/projected/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-kube-api-access-dhhzc\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:56.931343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.931198 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:56.931343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.931216 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhw8m\" (UniqueName: \"kubernetes.io/projected/32c19abf-04cd-4b8c-8edc-88fa7c0bddda-kube-api-access-fhw8m\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:56.931343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:56.931229 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f58bd3-f5bc-4b96-9203-9bfb637d58ee-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:57.344680 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.344638 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" containerID="cri-o://2cae899439c1960be425c0df9485d1d16862d554441125de606cc3dd5d817b7b" gracePeriod=45 Nov 03 19:51:57.500506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.500474 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" Nov 03 19:51:57.500685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.500472 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx" event={"ID":"68f58bd3-f5bc-4b96-9203-9bfb637d58ee","Type":"ContainerDied","Data":"123972c841835d4dd3536a215b5e4108b7a22ad48c10d8ffbf79fcc68342c407"} Nov 03 19:51:57.500685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.500645 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="123972c841835d4dd3536a215b5e4108b7a22ad48c10d8ffbf79fcc68342c407" Nov 03 19:51:57.503157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.502189 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" event={"ID":"32c19abf-04cd-4b8c-8edc-88fa7c0bddda","Type":"ContainerDied","Data":"f3b02ccfde340505932e56d2710240a8cdea33dde876e009a1fa079a00c0c467"} Nov 03 19:51:57.503157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.502224 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b02ccfde340505932e56d2710240a8cdea33dde876e009a1fa079a00c0c467" Nov 03 19:51:57.503157 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.502275 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x" Nov 03 19:51:57.511424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.510182 5233 generic.go:356] "Generic (PLEG): container finished" podID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerID="2cae899439c1960be425c0df9485d1d16862d554441125de606cc3dd5d817b7b" exitCode=0 Nov 03 19:51:57.511424 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.510389 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" event={"ID":"ab25e7c7-0b58-48ec-be0e-51e8329c040f","Type":"ContainerDied","Data":"2cae899439c1960be425c0df9485d1d16862d554441125de606cc3dd5d817b7b"} Nov 03 19:51:57.623285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.623256 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.758854 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw\" (UniqueName: \"kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw\") pod \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.758940 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache\") pod \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.759439 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache" pod "ab25e7c7-0b58-48ec-be0e-51e8329c040f" (UID: "ab25e7c7-0b58-48ec-be0e-51e8329c040f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.759806 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs\") pod \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.759900 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home\") pod \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.759937 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm\") pod \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.760023 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir\") pod \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\" (UID: \"ab25e7c7-0b58-48ec-be0e-51e8329c040f\") " Nov 03 19:51:57.761019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.760734 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:57.783262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.761076 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir" pod "ab25e7c7-0b58-48ec-be0e-51e8329c040f" (UID: "ab25e7c7-0b58-48ec-be0e-51e8329c040f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:57.783262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.761249 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home" pod "ab25e7c7-0b58-48ec-be0e-51e8329c040f" (UID: "ab25e7c7-0b58-48ec-be0e-51e8329c040f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:57.783262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.764069 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs" pod "ab25e7c7-0b58-48ec-be0e-51e8329c040f" (UID: "ab25e7c7-0b58-48ec-be0e-51e8329c040f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:57.783262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.771289 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm" pod "ab25e7c7-0b58-48ec-be0e-51e8329c040f" (UID: "ab25e7c7-0b58-48ec-be0e-51e8329c040f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:51:57.814245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.795176 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw" pod "ab25e7c7-0b58-48ec-be0e-51e8329c040f" (UID: "ab25e7c7-0b58-48ec-be0e-51e8329c040f"). InnerVolumeSpecName "kube-api-access-5kvrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:57.861764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.861735 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvrw\" (UniqueName: \"kubernetes.io/projected/ab25e7c7-0b58-48ec-be0e-51e8329c040f-kube-api-access-5kvrw\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:57.861950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.861767 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:57.861950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.861783 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:57.861950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.861796 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:57.861950 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:57.861808 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab25e7c7-0b58-48ec-be0e-51e8329c040f-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:58.048795 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.048758 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d"] Nov 03 19:51:58.049281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.049252 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" containerID="cri-o://a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899" gracePeriod=30 Nov 03 19:51:58.521212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.521185 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" Nov 03 19:51:58.521419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.521232 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv" event={"ID":"ab25e7c7-0b58-48ec-be0e-51e8329c040f","Type":"ContainerDied","Data":"026ff76129a5c4d362d852a8d0f0ef6cf50455f3f5ba350645768bd220e7ecae"} Nov 03 19:51:58.521419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.521293 5233 scope.go:122] "RemoveContainer" containerID="2cae899439c1960be425c0df9485d1d16862d554441125de606cc3dd5d817b7b" Nov 03 19:51:58.544700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.544670 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv"] Nov 03 19:51:58.548902 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.548878 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2c3fc4/simulator-w1-2c3fc4-kserve-677f87c447-tlxrv"] Nov 03 19:51:58.606641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.606613 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8"] Nov 03 19:51:58.607301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607286 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f24de34e-fb61-4398-b65f-7d2c552a91a6" containerName="payload-processing" Nov 03 19:51:58.607301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607302 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24de34e-fb61-4398-b65f-7d2c552a91a6" containerName="payload-processing" Nov 03 19:51:58.607408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607316 5233 cpu_manager.go:414] "containerMap: removing container" podUID="32c19abf-04cd-4b8c-8edc-88fa7c0bddda" containerName="revoke-keys" Nov 03 19:51:58.607408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607321 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c19abf-04cd-4b8c-8edc-88fa7c0bddda" containerName="revoke-keys" Nov 03 19:51:58.607408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607355 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" Nov 03 19:51:58.607408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607362 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" Nov 03 19:51:58.607408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607369 5233 cpu_manager.go:414] "containerMap: removing container" podUID="68f58bd3-f5bc-4b96-9203-9bfb637d58ee" containerName="revoke-keys" Nov 03 19:51:58.607408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607373 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f58bd3-f5bc-4b96-9203-9bfb637d58ee" containerName="revoke-keys" Nov 03 19:51:58.607598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607470 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24de34e-fb61-4398-b65f-7d2c552a91a6" containerName="payload-processing" Nov 03 19:51:58.607598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607479 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c19abf-04cd-4b8c-8edc-88fa7c0bddda" containerName="revoke-keys" Nov 03 19:51:58.607598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607490 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" containerName="main" Nov 03 19:51:58.607598 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.607496 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f58bd3-f5bc-4b96-9203-9bfb637d58ee" containerName="revoke-keys" Nov 03 19:51:58.612367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.612351 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.623854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.623826 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:58.623991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.623918 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:51:58.643001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.638906 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8"] Nov 03 19:51:58.678736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.678698 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.678924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.678768 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.678924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.678893 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88xv\" (UniqueName: \"kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.780406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.780309 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.780406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.780365 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.780962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.780421 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88xv\" (UniqueName: \"kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.780962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.780940 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.782746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.782720 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.797399 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.797379 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88xv\" (UniqueName: \"kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:58.927705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:58.927234 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:51:59.243331 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:51:59.243297 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11baa5c_c733_4ae4_94e1_d6a40619a0e5.slice/crio-2b18a80b0f734a6af85037b1d1c464f04fc535fa806f5cec4d01f078aa204651 WatchSource:0}: Error finding container 2b18a80b0f734a6af85037b1d1c464f04fc535fa806f5cec4d01f078aa204651: Status 404 returned error can't find the container with id 2b18a80b0f734a6af85037b1d1c464f04fc535fa806f5cec4d01f078aa204651 Nov 03 19:51:59.243541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.243520 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8"] Nov 03 19:51:59.297561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.297528 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" podUID="b5318a5a-001a-4e66-a71c-a976de821d56" containerName="payload-processing" containerID="cri-o://ac7f5cbf04522ab9fb6f87a2a2fc28c7203a76390a369ee9899f37d1a5b75afe" gracePeriod=25 Nov 03 19:51:59.548126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.548026 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" event={"ID":"d11baa5c-c733-4ae4-94e1-d6a40619a0e5","Type":"ContainerStarted","Data":"2b18a80b0f734a6af85037b1d1c464f04fc535fa806f5cec4d01f078aa204651"} Nov 03 19:51:59.550336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.550310 5233 generic.go:356] "Generic (PLEG): container finished" podID="b5318a5a-001a-4e66-a71c-a976de821d56" containerID="ac7f5cbf04522ab9fb6f87a2a2fc28c7203a76390a369ee9899f37d1a5b75afe" exitCode=0 Nov 03 19:51:59.550481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.550370 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" event={"ID":"b5318a5a-001a-4e66-a71c-a976de821d56","Type":"ContainerDied","Data":"ac7f5cbf04522ab9fb6f87a2a2fc28c7203a76390a369ee9899f37d1a5b75afe"} Nov 03 19:51:59.671709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.670723 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" podUID="8aeac680-38eb-4ee0-8173-fe496a7a0b11" containerName="payload-processing" containerID="cri-o://5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29" gracePeriod=25 Nov 03 19:51:59.784350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.762647 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:51:59.850102 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.850071 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume\") pod \"b5318a5a-001a-4e66-a71c-a976de821d56\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " Nov 03 19:51:59.850641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.850396 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg\" (UniqueName: \"kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg\") pod \"b5318a5a-001a-4e66-a71c-a976de821d56\" (UID: \"b5318a5a-001a-4e66-a71c-a976de821d56\") " Nov 03 19:51:59.850641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.850583 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume" pod "b5318a5a-001a-4e66-a71c-a976de821d56" (UID: "b5318a5a-001a-4e66-a71c-a976de821d56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:59.851138 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.851102 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5318a5a-001a-4e66-a71c-a976de821d56-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:59.870141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.869319 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg" pod "b5318a5a-001a-4e66-a71c-a976de821d56" (UID: "b5318a5a-001a-4e66-a71c-a976de821d56"). InnerVolumeSpecName "kube-api-access-8pwbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:59.873158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.873136 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab25e7c7-0b58-48ec-be0e-51e8329c040f" path="/var/lib/kubelet/pods/ab25e7c7-0b58-48ec-be0e-51e8329c040f/volumes" Nov 03 19:51:59.899291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.899263 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv"] Nov 03 19:51:59.904933 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.904689 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ce34d-blzfv"] Nov 03 19:51:59.953029 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:51:59.952286 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwbg\" (UniqueName: \"kubernetes.io/projected/b5318a5a-001a-4e66-a71c-a976de821d56-kube-api-access-8pwbg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:00.142909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.142876 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:52:00.280757 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.280725 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume\") pod \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " Nov 03 19:52:00.280932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.280804 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl\" (UniqueName: \"kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl\") pod \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\" (UID: \"8aeac680-38eb-4ee0-8173-fe496a7a0b11\") " Nov 03 19:52:00.281109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.281084 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume" pod "8aeac680-38eb-4ee0-8173-fe496a7a0b11" (UID: "8aeac680-38eb-4ee0-8173-fe496a7a0b11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:00.281304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.281287 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeac680-38eb-4ee0-8173-fe496a7a0b11-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:00.282754 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.282726 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl" pod "8aeac680-38eb-4ee0-8173-fe496a7a0b11" (UID: "8aeac680-38eb-4ee0-8173-fe496a7a0b11"). InnerVolumeSpecName "kube-api-access-74kjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:00.382741 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.382544 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74kjl\" (UniqueName: \"kubernetes.io/projected/8aeac680-38eb-4ee0-8173-fe496a7a0b11-kube-api-access-74kjl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:00.560078 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.560036 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" Nov 03 19:52:00.560285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.560089 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq" event={"ID":"b5318a5a-001a-4e66-a71c-a976de821d56","Type":"ContainerDied","Data":"5329a343e8ef55b3fedf8e233b630b04248aac221e6bb872b886e1bb7319a622"} Nov 03 19:52:00.560285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.560159 5233 scope.go:122] "RemoveContainer" containerID="ac7f5cbf04522ab9fb6f87a2a2fc28c7203a76390a369ee9899f37d1a5b75afe" Nov 03 19:52:00.561657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.561632 5233 generic.go:356] "Generic (PLEG): container finished" podID="8aeac680-38eb-4ee0-8173-fe496a7a0b11" containerID="5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29" exitCode=0 Nov 03 19:52:00.561793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.561682 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" Nov 03 19:52:00.561909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.561886 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" event={"ID":"8aeac680-38eb-4ee0-8173-fe496a7a0b11","Type":"ContainerDied","Data":"5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29"} Nov 03 19:52:00.562097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.561940 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx" event={"ID":"8aeac680-38eb-4ee0-8173-fe496a7a0b11","Type":"ContainerDied","Data":"034c9cdee03ec5d9fc8839b255c97b4bda1fe29c8a9cc30a3d4b9209f3a03acc"} Nov 03 19:52:00.564381 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.564353 5233 generic.go:356] "Generic (PLEG): container finished" podID="d11baa5c-c733-4ae4-94e1-d6a40619a0e5" containerID="25b64575cd60217070363a79bb6aa566f364a8fb91cf79f4ceac03775ed549d8" exitCode=0 Nov 03 19:52:00.564481 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.564415 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" event={"ID":"d11baa5c-c733-4ae4-94e1-d6a40619a0e5","Type":"ContainerDied","Data":"25b64575cd60217070363a79bb6aa566f364a8fb91cf79f4ceac03775ed549d8"} Nov 03 19:52:00.572913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.572891 5233 scope.go:122] "RemoveContainer" containerID="5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29" Nov 03 19:52:00.582903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.582884 5233 scope.go:122] "RemoveContainer" containerID="5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29" Nov 03 19:52:00.583185 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:00.583165 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29\": container with ID starting with 5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29 not found: ID does not exist" containerID="5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29" Nov 03 19:52:00.583251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.583191 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29"} err="failed to get container status \"5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29\": rpc error: code = NotFound desc = could not find container \"5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29\": container with ID starting with 5036b2f84e30818463059a8d42e822e07e8f732b7771bf85f2d66fe8a3468b29 not found: ID does not exist" Nov 03 19:52:00.608162 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.608140 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq"] Nov 03 19:52:00.615281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.615260 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab584a-75597bf695-kj4zq"] Nov 03 19:52:00.628360 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.628162 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx"] Nov 03 19:52:00.633234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:00.633186 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-29ab98-b65f6f8c9-kvcwx"] Nov 03 19:52:01.787028 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.786991 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:52:01.800541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.800518 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca\") pod \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " Nov 03 19:52:01.800681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.800611 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv\" (UniqueName: \"kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv\") pod \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " Nov 03 19:52:01.800750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.800682 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca\") pod \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\" (UID: \"d11baa5c-c733-4ae4-94e1-d6a40619a0e5\") " Nov 03 19:52:01.800916 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.800893 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca" pod "d11baa5c-c733-4ae4-94e1-d6a40619a0e5" (UID: "d11baa5c-c733-4ae4-94e1-d6a40619a0e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:01.801331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.801315 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:01.802705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.802681 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv" pod "d11baa5c-c733-4ae4-94e1-d6a40619a0e5" (UID: "d11baa5c-c733-4ae4-94e1-d6a40619a0e5"). InnerVolumeSpecName "kube-api-access-b88xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:01.802705 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.802695 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca" pod "d11baa5c-c733-4ae4-94e1-d6a40619a0e5" (UID: "d11baa5c-c733-4ae4-94e1-d6a40619a0e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:01.840596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.840563 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e07716f-cfcc-4b72-b294-96b3d2da394b" path="/var/lib/kubelet/pods/6e07716f-cfcc-4b72-b294-96b3d2da394b/volumes" Nov 03 19:52:01.845180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.841235 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aeac680-38eb-4ee0-8173-fe496a7a0b11" path="/var/lib/kubelet/pods/8aeac680-38eb-4ee0-8173-fe496a7a0b11/volumes" Nov 03 19:52:01.845180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.841793 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5318a5a-001a-4e66-a71c-a976de821d56" path="/var/lib/kubelet/pods/b5318a5a-001a-4e66-a71c-a976de821d56/volumes" Nov 03 19:52:01.902765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.902740 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88xv\" (UniqueName: \"kubernetes.io/projected/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-kube-api-access-b88xv\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:01.902765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:01.902762 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11baa5c-c733-4ae4-94e1-d6a40619a0e5-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:02.613658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:02.613626 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" Nov 03 19:52:02.613861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:02.613659 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8" event={"ID":"d11baa5c-c733-4ae4-94e1-d6a40619a0e5","Type":"ContainerDied","Data":"2b18a80b0f734a6af85037b1d1c464f04fc535fa806f5cec4d01f078aa204651"} Nov 03 19:52:02.613861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:02.613736 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b18a80b0f734a6af85037b1d1c464f04fc535fa806f5cec4d01f078aa204651" Nov 03 19:52:03.446115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.446085 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:52:03.541363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.541330 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.541562 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.541440 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.541720 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542365 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542407 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542449 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542473 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542738 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542817 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.543302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.542848 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket\") pod \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\" (UID: \"8ed3f885-db2f-47e6-abe0-f45a2938e66f\") " Nov 03 19:52:03.546526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.543467 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.546526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.543668 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:03.546526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.544824 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:03.546526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.545128 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:03.546526 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.545180 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:03.548913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.548888 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:03.549190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.549170 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:03.549457 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.549416 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:03.549558 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.549495 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw" pod "8ed3f885-db2f-47e6-abe0-f45a2938e66f" (UID: "8ed3f885-db2f-47e6-abe0-f45a2938e66f"). InnerVolumeSpecName "kube-api-access-sbthw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:03.623578 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.623545 5233 generic.go:356] "Generic (PLEG): container finished" podID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerID="a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899" exitCode=0 Nov 03 19:52:03.623770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.623603 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" Nov 03 19:52:03.623770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.623630 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" event={"ID":"8ed3f885-db2f-47e6-abe0-f45a2938e66f","Type":"ContainerDied","Data":"a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899"} Nov 03 19:52:03.623770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.623688 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d" event={"ID":"8ed3f885-db2f-47e6-abe0-f45a2938e66f","Type":"ContainerDied","Data":"221ed7bc5f16b2dadbb7f2429067db2a1279bdd1ff53d136b2575c4f85fd3e81"} Nov 03 19:52:03.623770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.623712 5233 scope.go:122] "RemoveContainer" containerID="a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899" Nov 03 19:52:03.635897 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.635880 5233 scope.go:122] "RemoveContainer" containerID="a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899" Nov 03 19:52:03.636174 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:03.636154 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899\": container with ID starting with a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899 not found: ID does not exist" containerID="a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899" Nov 03 19:52:03.636241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.636182 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899"} err="failed to get container status \"a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899\": rpc error: code = NotFound desc = could not find container \"a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899\": container with ID starting with a1157a9d85218a6301f83075b27da1d372032ca97194011cee7004bbb9123899 not found: ID does not exist" Nov 03 19:52:03.645585 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645559 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645585 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645584 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645602 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645616 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645627 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645636 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645648 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbthw\" (UniqueName: \"kubernetes.io/projected/8ed3f885-db2f-47e6-abe0-f45a2938e66f-kube-api-access-sbthw\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.645839 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.645661 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed3f885-db2f-47e6-abe0-f45a2938e66f-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:03.654051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.654027 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d"] Nov 03 19:52:03.660927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.660871 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c54f85-openshift-default-5d756c69f5-ntw6d"] Nov 03 19:52:03.841485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.841452 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" path="/var/lib/kubelet/pods/8ed3f885-db2f-47e6-abe0-f45a2938e66f/volumes" Nov 03 19:52:03.948134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.937509 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk"] Nov 03 19:52:03.948134 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:03.943377 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29ab98-nwkwk"] Nov 03 19:52:05.848678 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:05.848591 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbaf7870-eca1-4b49-b3b1-ceffe8a381f4" path="/var/lib/kubelet/pods/bbaf7870-eca1-4b49-b3b1-ceffe8a381f4/volumes" Nov 03 19:52:06.797865 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:06.797785 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7"] Nov 03 19:52:06.821247 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:06.821125 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2c3fc4-xp2c7"] Nov 03 19:52:07.839696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:07.839661 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f57fd5-cbfc-4608-84b6-0ff8f0c87270" path="/var/lib/kubelet/pods/c9f57fd5-cbfc-4608-84b6-0ff8f0c87270/volumes" Nov 03 19:52:09.420924 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.420878 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk"] Nov 03 19:52:09.421946 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.421922 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b5318a5a-001a-4e66-a71c-a976de821d56" containerName="payload-processing" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.421946 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5318a5a-001a-4e66-a71c-a976de821d56" containerName="payload-processing" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.421961 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8aeac680-38eb-4ee0-8173-fe496a7a0b11" containerName="payload-processing" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.421985 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aeac680-38eb-4ee0-8173-fe496a7a0b11" containerName="payload-processing" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422022 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422030 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422044 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d11baa5c-c733-4ae4-94e1-d6a40619a0e5" containerName="revoke-keys" Nov 03 19:52:09.422073 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422051 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11baa5c-c733-4ae4-94e1-d6a40619a0e5" containerName="revoke-keys" Nov 03 19:52:09.422477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422197 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11baa5c-c733-4ae4-94e1-d6a40619a0e5" containerName="revoke-keys" Nov 03 19:52:09.422477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422218 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5318a5a-001a-4e66-a71c-a976de821d56" containerName="payload-processing" Nov 03 19:52:09.422477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422228 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aeac680-38eb-4ee0-8173-fe496a7a0b11" containerName="payload-processing" Nov 03 19:52:09.422477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.422244 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed3f885-db2f-47e6-abe0-f45a2938e66f" containerName="istio-proxy" Nov 03 19:52:09.434228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.434192 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk"] Nov 03 19:52:09.434494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.434317 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.446044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.446013 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:09.458023 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.450628 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:52:09.599921 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.599880 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpmh\" (UniqueName: \"kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.600140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.600119 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.600219 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.600204 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.701337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.701250 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.701521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.701369 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.701521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.701479 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpmh\" (UniqueName: \"kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.702280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.702253 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.703883 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.703861 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.713942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.713920 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpmh\" (UniqueName: \"kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh\") pod \"maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:09.766066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:09.766037 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:10.071133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:10.070921 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk"] Nov 03 19:52:10.077814 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:10.077778 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b5da44_cf9c_4474_b1f9_c841db16285e.slice/crio-91aba3f8558e2f177b3a658bb8fd372a0523b9dcfd4f08584df54ddf00e790c8 WatchSource:0}: Error finding container 91aba3f8558e2f177b3a658bb8fd372a0523b9dcfd4f08584df54ddf00e790c8: Status 404 returned error can't find the container with id 91aba3f8558e2f177b3a658bb8fd372a0523b9dcfd4f08584df54ddf00e790c8 Nov 03 19:52:11.044915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:11.043267 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" event={"ID":"b0b5da44-cf9c-4474-b1f9-c841db16285e","Type":"ContainerStarted","Data":"8229fbfb67fb4f7c6fd15082fd33722975ce8035cb3a038da44e96e14cd2d6e7"} Nov 03 19:52:11.044915 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:11.043384 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" event={"ID":"b0b5da44-cf9c-4474-b1f9-c841db16285e","Type":"ContainerStarted","Data":"91aba3f8558e2f177b3a658bb8fd372a0523b9dcfd4f08584df54ddf00e790c8"} Nov 03 19:52:11.066266 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:11.066188 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" podStartSLOduration=2.06616427 podStartE2EDuration="2.06616427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:11.06338094 +0000 UTC m=+1497.404494440" watchObservedRunningTime="2026-11-03 19:52:11.06616427 +0000 UTC m=+1497.407277748" Nov 03 19:52:12.027015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.026959 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64"] Nov 03 19:52:12.033263 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.033229 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab584a-rdc64"] Nov 03 19:52:12.053319 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.053284 5233 generic.go:356] "Generic (PLEG): container finished" podID="b0b5da44-cf9c-4474-b1f9-c841db16285e" containerID="8229fbfb67fb4f7c6fd15082fd33722975ce8035cb3a038da44e96e14cd2d6e7" exitCode=0 Nov 03 19:52:12.053697 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.053333 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" event={"ID":"b0b5da44-cf9c-4474-b1f9-c841db16285e","Type":"ContainerDied","Data":"8229fbfb67fb4f7c6fd15082fd33722975ce8035cb3a038da44e96e14cd2d6e7"} Nov 03 19:52:12.477609 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.477579 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n"] Nov 03 19:52:12.481799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.481778 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.531967 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.531923 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n"] Nov 03 19:52:12.605428 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.605389 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.605634 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.605477 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.605834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.605801 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxs4\" (UniqueName: \"kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.708054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.707947 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.708230 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.708066 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.708313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.708267 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxs4\" (UniqueName: \"kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.709841 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.709812 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.712733 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.712682 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.723355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.723328 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxs4\" (UniqueName: \"kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:12.816830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:12.816747 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:13.128871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.128831 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n"] Nov 03 19:52:13.302756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.302733 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:13.421904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.421870 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca\") pod \"b0b5da44-cf9c-4474-b1f9-c841db16285e\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " Nov 03 19:52:13.422100 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.422013 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh\" (UniqueName: \"kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh\") pod \"b0b5da44-cf9c-4474-b1f9-c841db16285e\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " Nov 03 19:52:13.422182 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.422164 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca\") pod \"b0b5da44-cf9c-4474-b1f9-c841db16285e\" (UID: \"b0b5da44-cf9c-4474-b1f9-c841db16285e\") " Nov 03 19:52:13.423510 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.423480 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca" pod "b0b5da44-cf9c-4474-b1f9-c841db16285e" (UID: "b0b5da44-cf9c-4474-b1f9-c841db16285e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:13.425729 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.425696 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca" pod "b0b5da44-cf9c-4474-b1f9-c841db16285e" (UID: "b0b5da44-cf9c-4474-b1f9-c841db16285e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:13.426055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.426036 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh" pod "b0b5da44-cf9c-4474-b1f9-c841db16285e" (UID: "b0b5da44-cf9c-4474-b1f9-c841db16285e"). InnerVolumeSpecName "kube-api-access-hjpmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:13.524679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.524645 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:13.524679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.524674 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b5da44-cf9c-4474-b1f9-c841db16285e-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:13.524679 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.524684 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjpmh\" (UniqueName: \"kubernetes.io/projected/b0b5da44-cf9c-4474-b1f9-c841db16285e-kube-api-access-hjpmh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:13.842749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:13.842714 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a41fcc-0605-486e-b1d5-3315d844c0ce" path="/var/lib/kubelet/pods/87a41fcc-0605-486e-b1d5-3315d844c0ce/volumes" Nov 03 19:52:14.078630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:14.078580 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" event={"ID":"fe817ae7-92d5-4ce4-adcd-06992347c50c","Type":"ContainerStarted","Data":"ef6833943cd041da0e52e034d877fc35707577d7752e7f6ee8093bcb8ca861eb"} Nov 03 19:52:14.078868 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:14.078652 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" event={"ID":"fe817ae7-92d5-4ce4-adcd-06992347c50c","Type":"ContainerStarted","Data":"71527bae22be928fd46a925275d628dcf8092fbf03a16d2b6bea2a671dc5d91e"} Nov 03 19:52:14.080830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:14.080724 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" Nov 03 19:52:14.080830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:14.080755 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk" event={"ID":"b0b5da44-cf9c-4474-b1f9-c841db16285e","Type":"ContainerDied","Data":"91aba3f8558e2f177b3a658bb8fd372a0523b9dcfd4f08584df54ddf00e790c8"} Nov 03 19:52:14.080830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:14.080815 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91aba3f8558e2f177b3a658bb8fd372a0523b9dcfd4f08584df54ddf00e790c8" Nov 03 19:52:14.107579 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:14.107345 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" podStartSLOduration=2.107317967 podStartE2EDuration="2.107317967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:14.101731079 +0000 UTC m=+1500.442844565" watchObservedRunningTime="2026-11-03 19:52:14.107317967 +0000 UTC m=+1500.448431453" Nov 03 19:52:15.093218 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:15.093182 5233 generic.go:356] "Generic (PLEG): container finished" podID="fe817ae7-92d5-4ce4-adcd-06992347c50c" containerID="ef6833943cd041da0e52e034d877fc35707577d7752e7f6ee8093bcb8ca861eb" exitCode=0 Nov 03 19:52:15.093715 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:15.093267 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" event={"ID":"fe817ae7-92d5-4ce4-adcd-06992347c50c","Type":"ContainerDied","Data":"ef6833943cd041da0e52e034d877fc35707577d7752e7f6ee8093bcb8ca861eb"} Nov 03 19:52:16.286146 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.286121 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:16.378960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.378873 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca\") pod \"fe817ae7-92d5-4ce4-adcd-06992347c50c\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " Nov 03 19:52:16.379143 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.378986 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca\") pod \"fe817ae7-92d5-4ce4-adcd-06992347c50c\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " Nov 03 19:52:16.379252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.379236 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4\" (UniqueName: \"kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4\") pod \"fe817ae7-92d5-4ce4-adcd-06992347c50c\" (UID: \"fe817ae7-92d5-4ce4-adcd-06992347c50c\") " Nov 03 19:52:16.379408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.379384 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca" pod "fe817ae7-92d5-4ce4-adcd-06992347c50c" (UID: "fe817ae7-92d5-4ce4-adcd-06992347c50c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:16.379761 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.379740 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:16.381799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.381774 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca" pod "fe817ae7-92d5-4ce4-adcd-06992347c50c" (UID: "fe817ae7-92d5-4ce4-adcd-06992347c50c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:16.381907 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.381787 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4" pod "fe817ae7-92d5-4ce4-adcd-06992347c50c" (UID: "fe817ae7-92d5-4ce4-adcd-06992347c50c"). InnerVolumeSpecName "kube-api-access-nxxs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:16.402791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.402762 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8"] Nov 03 19:52:16.403044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.403012 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" containerID="cri-o://3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d" gracePeriod=30 Nov 03 19:52:16.409639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.409613 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc"] Nov 03 19:52:16.409856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.409834 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" containerID="cri-o://c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52" gracePeriod=30 Nov 03 19:52:16.518997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.518672 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe817ae7-92d5-4ce4-adcd-06992347c50c-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:16.518997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:16.518711 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxs4\" (UniqueName: \"kubernetes.io/projected/fe817ae7-92d5-4ce4-adcd-06992347c50c-kube-api-access-nxxs4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:17.132663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:17.132634 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" Nov 03 19:52:17.132663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:17.132638 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n" event={"ID":"fe817ae7-92d5-4ce4-adcd-06992347c50c","Type":"ContainerDied","Data":"71527bae22be928fd46a925275d628dcf8092fbf03a16d2b6bea2a671dc5d91e"} Nov 03 19:52:17.132855 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:17.132693 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71527bae22be928fd46a925275d628dcf8092fbf03a16d2b6bea2a671dc5d91e" Nov 03 19:52:17.251374 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:17.251343 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8"] Nov 03 19:52:17.256148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:17.256121 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2b2e60-rlqd8"] Nov 03 19:52:17.839774 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:17.839737 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11baa5c-c733-4ae4-94e1-d6a40619a0e5" path="/var/lib/kubelet/pods/d11baa5c-c733-4ae4-94e1-d6a40619a0e5/volumes" Nov 03 19:52:19.303009 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:19.302935 5233 patch_prober.go:29] interesting pod/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:52:19.303376 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:19.303011 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:52:19.551919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:19.551886 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz"] Nov 03 19:52:19.565204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:19.565120 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7"] Nov 03 19:52:21.278156 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:21.278119 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:52:21.463050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.463020 5233 scope.go:122] "RemoveContainer" containerID="2227682ca58334f14202439ae5a1f443065e237133b529ffe51e160b9c8dce2e" Nov 03 19:52:21.655724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.655237 5233 scope.go:122] "RemoveContainer" containerID="e24e0add43b38746fc8b81ed0790aadb2ac7e3e6ec963c9d72304f81ed0f2891" Nov 03 19:52:21.895989 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.895947 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:52:21.935115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.935082 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:52:21.942012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.941963 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.942148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942057 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.942225 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942155 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.942283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942227 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.942283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942272 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.942398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942324 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.942398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942339 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.942398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942373 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.942567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942401 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.942567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942446 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.942567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942476 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.942567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942501 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.942567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942530 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.942567 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942561 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942606 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942712 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942720 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942749 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942790 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942845 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs\") pod \"cb36251e-85de-4139-ac16-d0642a2bbace\" (UID: \"cb36251e-85de-4139-ac16-d0642a2bbace\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.942902 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943013 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket\") pod \"388328ea-4202-411e-baae-80d3c6e875a3\" (UID: \"388328ea-4202-411e-baae-80d3c6e875a3\") " Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943281 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943524 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943874 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943897 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943922 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943935 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb36251e-85de-4139-ac16-d0642a2bbace-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.943952 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.944006 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.946391 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.947130 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.947190 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.948728 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.948962 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.951597 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "kube-api-access-66zbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.951650 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "kube-api-access-dxzf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:21.953061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.952876 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.953963 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.953502 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo" pod "388328ea-4202-411e-baae-80d3c6e875a3" (UID: "388328ea-4202-411e-baae-80d3c6e875a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:21.953963 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.953920 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:21.956213 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.955180 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:21.956458 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:21.956436 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo" pod "cb36251e-85de-4139-ac16-d0642a2bbace" (UID: "cb36251e-85de-4139-ac16-d0642a2bbace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:22.044834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044800 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb36251e-85de-4139-ac16-d0642a2bbace-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.044834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044830 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388328ea-4202-411e-baae-80d3c6e875a3-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.044834 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044841 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044852 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzf4\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-kube-api-access-dxzf4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044863 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044871 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044880 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388328ea-4202-411e-baae-80d3c6e875a3-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044888 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66zbd\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-kube-api-access-66zbd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044896 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388328ea-4202-411e-baae-80d3c6e875a3-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044904 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb36251e-85de-4139-ac16-d0642a2bbace-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044912 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb36251e-85de-4139-ac16-d0642a2bbace-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.045099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.044920 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388328ea-4202-411e-baae-80d3c6e875a3-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:22.105681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.105608 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h"] Nov 03 19:52:22.106257 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106242 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b0b5da44-cf9c-4474-b1f9-c841db16285e" containerName="revoke-keys" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106257 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b5da44-cf9c-4474-b1f9-c841db16285e" containerName="revoke-keys" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106273 5233 cpu_manager.go:414] "containerMap: removing container" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106278 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106285 5233 cpu_manager.go:414] "containerMap: removing container" podUID="fe817ae7-92d5-4ce4-adcd-06992347c50c" containerName="revoke-keys" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106292 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe817ae7-92d5-4ce4-adcd-06992347c50c" containerName="revoke-keys" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106310 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" Nov 03 19:52:22.106321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106315 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" Nov 03 19:52:22.106530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106337 5233 cpu_manager.go:414] "containerMap: removing container" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" Nov 03 19:52:22.106530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106342 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" Nov 03 19:52:22.106530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106423 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b5da44-cf9c-4474-b1f9-c841db16285e" containerName="revoke-keys" Nov 03 19:52:22.106530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106432 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" Nov 03 19:52:22.106530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106438 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe817ae7-92d5-4ce4-adcd-06992347c50c" containerName="revoke-keys" Nov 03 19:52:22.106530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.106445 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" containerName="istio-proxy" Nov 03 19:52:22.111383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.111363 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.119214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.119185 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h"] Nov 03 19:52:22.135183 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.135151 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:52:22.135310 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.135231 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:22.167656 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.150564 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.172548 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.172353 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.174041 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.173050 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tspkx\" (UniqueName: \"kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.199365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.199331 5233 generic.go:356] "Generic (PLEG): container finished" podID="cb36251e-85de-4139-ac16-d0642a2bbace" containerID="3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d" exitCode=0 Nov 03 19:52:22.199707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.199675 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" event={"ID":"cb36251e-85de-4139-ac16-d0642a2bbace","Type":"ContainerDied","Data":"3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d"} Nov 03 19:52:22.199909 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.199891 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" event={"ID":"cb36251e-85de-4139-ac16-d0642a2bbace","Type":"ContainerDied","Data":"32502ee6c964326d063a45368547b90c828340ac90ddfa982a596deb4c538c42"} Nov 03 19:52:22.200061 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.200041 5233 scope.go:122] "RemoveContainer" containerID="3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d" Nov 03 19:52:22.200383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.200359 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8" Nov 03 19:52:22.203735 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.203706 5233 generic.go:356] "Generic (PLEG): container finished" podID="388328ea-4202-411e-baae-80d3c6e875a3" containerID="c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52" exitCode=0 Nov 03 19:52:22.203872 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.203850 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" event={"ID":"388328ea-4202-411e-baae-80d3c6e875a3","Type":"ContainerDied","Data":"c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52"} Nov 03 19:52:22.203962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.203944 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" event={"ID":"388328ea-4202-411e-baae-80d3c6e875a3","Type":"ContainerDied","Data":"e49b3b209ade411180ec3b9dd9b28a9cd1676fca2733b9ba925aae20ba4863e5"} Nov 03 19:52:22.204075 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.204053 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc" Nov 03 19:52:22.218012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.217850 5233 scope.go:122] "RemoveContainer" containerID="3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d" Nov 03 19:52:22.218546 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:22.218262 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d\": container with ID starting with 3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d not found: ID does not exist" containerID="3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d" Nov 03 19:52:22.218546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.218296 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d"} err="failed to get container status \"3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d\": rpc error: code = NotFound desc = could not find container \"3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d\": container with ID starting with 3b189c6137adc0b2f1ee7a04984ec29a27b90fc5bbd6352d77aaf92cbe9c358d not found: ID does not exist" Nov 03 19:52:22.218546 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.218319 5233 scope.go:122] "RemoveContainer" containerID="c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52" Nov 03 19:52:22.230922 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.230903 5233 scope.go:122] "RemoveContainer" containerID="502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3" Nov 03 19:52:22.240249 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.240221 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8"] Nov 03 19:52:22.243527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.243503 5233 scope.go:122] "RemoveContainer" containerID="c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52" Nov 03 19:52:22.243904 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:22.243882 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52\": container with ID starting with c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52 not found: ID does not exist" containerID="c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52" Nov 03 19:52:22.244008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.243908 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52"} err="failed to get container status \"c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52\": rpc error: code = NotFound desc = could not find container \"c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52\": container with ID starting with c0a85259820218bace4dbba6a04397c52bc1c42dc5d3c95dec8f5c8604f47e52 not found: ID does not exist" Nov 03 19:52:22.244008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.243926 5233 scope.go:122] "RemoveContainer" containerID="502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3" Nov 03 19:52:22.244362 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:22.244341 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3\": container with ID starting with 502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3 not found: ID does not exist" containerID="502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3" Nov 03 19:52:22.244438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.244365 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3"} err="failed to get container status \"502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3\": rpc error: code = NotFound desc = could not find container \"502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3\": container with ID starting with 502b8371370f5aeb50a91bdf8ff1f8210824788b4978aa4f46cafa5aba009fe3 not found: ID does not exist" Nov 03 19:52:22.245119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.245100 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-p7bg8"] Nov 03 19:52:22.262292 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.262270 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc"] Nov 03 19:52:22.271669 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.271646 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2c3fc4-openshift-default-77ff649d77-xwsrc"] Nov 03 19:52:22.275860 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.275840 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tspkx\" (UniqueName: \"kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.275941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.275899 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.276066 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.276051 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.276663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.276636 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.278133 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.278114 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.289714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.289692 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tspkx\" (UniqueName: \"kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx\") pod \"maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.462087 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.462055 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:22.755494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:22.755473 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h"] Nov 03 19:52:22.758234 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:22.758202 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc527f6_c48e_42ac_8583_2965cb822685.slice/crio-d513bd24eab54c1811996fa025db1b1ece1ba4394128389143588f879cd41911 WatchSource:0}: Error finding container d513bd24eab54c1811996fa025db1b1ece1ba4394128389143588f879cd41911: Status 404 returned error can't find the container with id d513bd24eab54c1811996fa025db1b1ece1ba4394128389143588f879cd41911 Nov 03 19:52:23.228082 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:23.227662 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" event={"ID":"9cc527f6-c48e-42ac-8583-2965cb822685","Type":"ContainerStarted","Data":"d513bd24eab54c1811996fa025db1b1ece1ba4394128389143588f879cd41911"} Nov 03 19:52:23.860511 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:23.860475 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388328ea-4202-411e-baae-80d3c6e875a3" path="/var/lib/kubelet/pods/388328ea-4202-411e-baae-80d3c6e875a3/volumes" Nov 03 19:52:23.861451 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:23.861428 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb36251e-85de-4139-ac16-d0642a2bbace" path="/var/lib/kubelet/pods/cb36251e-85de-4139-ac16-d0642a2bbace/volumes" Nov 03 19:52:24.252277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.252241 5233 generic.go:356] "Generic (PLEG): container finished" podID="9cc527f6-c48e-42ac-8583-2965cb822685" containerID="3e2a82dbb08118792fe73ad0f3b1ef1610ce0c5d3b6801b18cdd626428ceb5f2" exitCode=0 Nov 03 19:52:24.252466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.252330 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" event={"ID":"9cc527f6-c48e-42ac-8583-2965cb822685","Type":"ContainerDied","Data":"3e2a82dbb08118792fe73ad0f3b1ef1610ce0c5d3b6801b18cdd626428ceb5f2"} Nov 03 19:52:24.552635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.552522 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" podUID="3c010d8b-f1a3-4c1b-83e5-119fc481188c" containerName="payload-processing" containerID="cri-o://948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917" gracePeriod=25 Nov 03 19:52:24.565769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.565739 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" podUID="16792b6b-0663-4689-a3fa-16e7418eed74" containerName="payload-pre-processing" containerID="cri-o://9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d" gracePeriod=25 Nov 03 19:52:24.853385 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.853345 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:52:24.928285 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.928250 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:52:24.934778 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.934736 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume\") pod \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " Nov 03 19:52:24.935045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.935025 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx\" (UniqueName: \"kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx\") pod \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\" (UID: \"3c010d8b-f1a3-4c1b-83e5-119fc481188c\") " Nov 03 19:52:24.935160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.935137 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume" pod "3c010d8b-f1a3-4c1b-83e5-119fc481188c" (UID: "3c010d8b-f1a3-4c1b-83e5-119fc481188c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:24.935797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.935776 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c010d8b-f1a3-4c1b-83e5-119fc481188c-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:24.937571 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:24.937543 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx" pod "3c010d8b-f1a3-4c1b-83e5-119fc481188c" (UID: "3c010d8b-f1a3-4c1b-83e5-119fc481188c"). InnerVolumeSpecName "kube-api-access-gqkgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:25.037298 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.037260 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs\" (UniqueName: \"kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs\") pod \"16792b6b-0663-4689-a3fa-16e7418eed74\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " Nov 03 19:52:25.037469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.037457 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume\") pod \"16792b6b-0663-4689-a3fa-16e7418eed74\" (UID: \"16792b6b-0663-4689-a3fa-16e7418eed74\") " Nov 03 19:52:25.037833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.037803 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume" pod "16792b6b-0663-4689-a3fa-16e7418eed74" (UID: "16792b6b-0663-4689-a3fa-16e7418eed74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:25.038099 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.038080 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16792b6b-0663-4689-a3fa-16e7418eed74-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:25.038193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.038104 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqkgx\" (UniqueName: \"kubernetes.io/projected/3c010d8b-f1a3-4c1b-83e5-119fc481188c-kube-api-access-gqkgx\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:25.039536 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.039510 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs" pod "16792b6b-0663-4689-a3fa-16e7418eed74" (UID: "16792b6b-0663-4689-a3fa-16e7418eed74"). InnerVolumeSpecName "kube-api-access-ghfzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:25.139151 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.139074 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghfzs\" (UniqueName: \"kubernetes.io/projected/16792b6b-0663-4689-a3fa-16e7418eed74-kube-api-access-ghfzs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:25.264198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.264161 5233 generic.go:356] "Generic (PLEG): container finished" podID="3c010d8b-f1a3-4c1b-83e5-119fc481188c" containerID="948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917" exitCode=0 Nov 03 19:52:25.264361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.264205 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" Nov 03 19:52:25.264361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.264243 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" event={"ID":"3c010d8b-f1a3-4c1b-83e5-119fc481188c","Type":"ContainerDied","Data":"948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917"} Nov 03 19:52:25.264361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.264294 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz" event={"ID":"3c010d8b-f1a3-4c1b-83e5-119fc481188c","Type":"ContainerDied","Data":"e84eb5ae1c9241b0bc4294fe8d606fc2a94180685ad317d7268aab573b26d5a4"} Nov 03 19:52:25.264361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.264320 5233 scope.go:122] "RemoveContainer" containerID="948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917" Nov 03 19:52:25.266049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.266025 5233 generic.go:356] "Generic (PLEG): container finished" podID="16792b6b-0663-4689-a3fa-16e7418eed74" containerID="9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d" exitCode=0 Nov 03 19:52:25.266158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.266084 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" Nov 03 19:52:25.266158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.266116 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" event={"ID":"16792b6b-0663-4689-a3fa-16e7418eed74","Type":"ContainerDied","Data":"9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d"} Nov 03 19:52:25.266236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.266171 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7" event={"ID":"16792b6b-0663-4689-a3fa-16e7418eed74","Type":"ContainerDied","Data":"d8a36666627629c1ac80f157a06d61618d7cb3cd9555c4f04d4b3f2b987ddf12"} Nov 03 19:52:25.277450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.277429 5233 scope.go:122] "RemoveContainer" containerID="948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917" Nov 03 19:52:25.277815 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:25.277781 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917\": container with ID starting with 948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917 not found: ID does not exist" containerID="948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917" Nov 03 19:52:25.277904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.277823 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917"} err="failed to get container status \"948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917\": rpc error: code = NotFound desc = could not find container \"948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917\": container with ID starting with 948aa7d39a2d8a4e97e73febeba7912b4124c02ba3b9f3c2ba56e29659132917 not found: ID does not exist" Nov 03 19:52:25.277904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.277846 5233 scope.go:122] "RemoveContainer" containerID="9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d" Nov 03 19:52:25.293821 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.293801 5233 scope.go:122] "RemoveContainer" containerID="9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d" Nov 03 19:52:25.294253 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:25.294232 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d\": container with ID starting with 9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d not found: ID does not exist" containerID="9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d" Nov 03 19:52:25.294357 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.294267 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d"} err="failed to get container status \"9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d\": rpc error: code = NotFound desc = could not find container \"9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d\": container with ID starting with 9e16a27f5615a7516afa03978828b93721681cf4b847fe96ac300ecd76ce900d not found: ID does not exist" Nov 03 19:52:25.306055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.306024 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n"] Nov 03 19:52:25.311115 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.311086 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-25944c-9sj6n"] Nov 03 19:52:25.315314 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.315285 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz"] Nov 03 19:52:25.319749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.319724 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6a08e-68dc66b4dc-rcnpz"] Nov 03 19:52:25.325661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.324826 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7"] Nov 03 19:52:25.329015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.328990 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6a08e-6f475c559f-jbrd7"] Nov 03 19:52:25.424217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.424188 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:25.557033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.545496 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx\" (UniqueName: \"kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx\") pod \"9cc527f6-c48e-42ac-8583-2965cb822685\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " Nov 03 19:52:25.559367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.559337 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca\") pod \"9cc527f6-c48e-42ac-8583-2965cb822685\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " Nov 03 19:52:25.560025 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.559959 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca\") pod \"9cc527f6-c48e-42ac-8583-2965cb822685\" (UID: \"9cc527f6-c48e-42ac-8583-2965cb822685\") " Nov 03 19:52:25.560640 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.559779 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca" pod "9cc527f6-c48e-42ac-8583-2965cb822685" (UID: "9cc527f6-c48e-42ac-8583-2965cb822685"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:25.564475 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.564448 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca" pod "9cc527f6-c48e-42ac-8583-2965cb822685" (UID: "9cc527f6-c48e-42ac-8583-2965cb822685"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:25.565891 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.565852 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:25.566065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.566042 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc527f6-c48e-42ac-8583-2965cb822685-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:25.570228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.570185 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx" pod "9cc527f6-c48e-42ac-8583-2965cb822685" (UID: "9cc527f6-c48e-42ac-8583-2965cb822685"). InnerVolumeSpecName "kube-api-access-tspkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:25.626593 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.626553 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s"] Nov 03 19:52:25.626867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.626835 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" containerID="cri-o://d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5" gracePeriod=30 Nov 03 19:52:25.667480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.667385 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tspkx\" (UniqueName: \"kubernetes.io/projected/9cc527f6-c48e-42ac-8583-2965cb822685-kube-api-access-tspkx\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:25.859515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.859480 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16792b6b-0663-4689-a3fa-16e7418eed74" path="/var/lib/kubelet/pods/16792b6b-0663-4689-a3fa-16e7418eed74/volumes" Nov 03 19:52:25.860749 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.860569 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c010d8b-f1a3-4c1b-83e5-119fc481188c" path="/var/lib/kubelet/pods/3c010d8b-f1a3-4c1b-83e5-119fc481188c/volumes" Nov 03 19:52:25.861817 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:25.861760 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe817ae7-92d5-4ce4-adcd-06992347c50c" path="/var/lib/kubelet/pods/fe817ae7-92d5-4ce4-adcd-06992347c50c/volumes" Nov 03 19:52:26.275799 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.275763 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" Nov 03 19:52:26.276283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.275800 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h" event={"ID":"9cc527f6-c48e-42ac-8583-2965cb822685","Type":"ContainerDied","Data":"d513bd24eab54c1811996fa025db1b1ece1ba4394128389143588f879cd41911"} Nov 03 19:52:26.276283 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.275859 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d513bd24eab54c1811996fa025db1b1ece1ba4394128389143588f879cd41911" Nov 03 19:52:26.344838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.344795 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7"] Nov 03 19:52:26.345733 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345710 5233 cpu_manager.go:414] "containerMap: removing container" podUID="16792b6b-0663-4689-a3fa-16e7418eed74" containerName="payload-pre-processing" Nov 03 19:52:26.345854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345733 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16792b6b-0663-4689-a3fa-16e7418eed74" containerName="payload-pre-processing" Nov 03 19:52:26.345854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345770 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9cc527f6-c48e-42ac-8583-2965cb822685" containerName="revoke-keys" Nov 03 19:52:26.345854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345777 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc527f6-c48e-42ac-8583-2965cb822685" containerName="revoke-keys" Nov 03 19:52:26.345854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345810 5233 cpu_manager.go:414] "containerMap: removing container" podUID="3c010d8b-f1a3-4c1b-83e5-119fc481188c" containerName="payload-processing" Nov 03 19:52:26.345854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345817 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c010d8b-f1a3-4c1b-83e5-119fc481188c" containerName="payload-processing" Nov 03 19:52:26.346068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.345986 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="16792b6b-0663-4689-a3fa-16e7418eed74" containerName="payload-pre-processing" Nov 03 19:52:26.346068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.346004 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc527f6-c48e-42ac-8583-2965cb822685" containerName="revoke-keys" Nov 03 19:52:26.346068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.346017 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="388328ea-4202-411e-baae-80d3c6e875a3" containerName="istio-proxy" Nov 03 19:52:26.346068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.346028 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c010d8b-f1a3-4c1b-83e5-119fc481188c" containerName="payload-processing" Nov 03 19:52:26.352228 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.352204 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.355398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.355372 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7"] Nov 03 19:52:26.357646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.357622 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:26.357789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.357716 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:52:26.478854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.478812 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7s7\" (UniqueName: \"kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.479387 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.479354 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.479521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.479431 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.580617 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.580534 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.580781 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.580662 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7s7\" (UniqueName: \"kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.580893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.580875 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.581277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.581246 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.583383 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.583361 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.593688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.593661 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7s7\" (UniqueName: \"kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7\") pod \"maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:26.676400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:26.676368 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:27.002435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:27.002408 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7"] Nov 03 19:52:27.003880 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:27.003852 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546cf8a8_7766_41dc_b7b6_a7396536b365.slice/crio-12728ad4309dc85bd3f6d11b48a7625c3785f4b2294741eba3d432637e0e697c WatchSource:0}: Error finding container 12728ad4309dc85bd3f6d11b48a7625c3785f4b2294741eba3d432637e0e697c: Status 404 returned error can't find the container with id 12728ad4309dc85bd3f6d11b48a7625c3785f4b2294741eba3d432637e0e697c Nov 03 19:52:27.288585 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:27.288550 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" event={"ID":"546cf8a8-7766-41dc-b7b6-a7396536b365","Type":"ContainerStarted","Data":"12728ad4309dc85bd3f6d11b48a7625c3785f4b2294741eba3d432637e0e697c"} Nov 03 19:52:28.301216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:28.301184 5233 generic.go:356] "Generic (PLEG): container finished" podID="546cf8a8-7766-41dc-b7b6-a7396536b365" containerID="4f29edc7b2c84e317b2d8634dc4b7edacfd0bdf4afd6f48d345f3eaee4b80e72" exitCode=0 Nov 03 19:52:28.301625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:28.301274 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" event={"ID":"546cf8a8-7766-41dc-b7b6-a7396536b365","Type":"ContainerDied","Data":"4f29edc7b2c84e317b2d8634dc4b7edacfd0bdf4afd6f48d345f3eaee4b80e72"} Nov 03 19:52:29.483853 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.483830 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:29.624137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.624048 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca\") pod \"546cf8a8-7766-41dc-b7b6-a7396536b365\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " Nov 03 19:52:29.624137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.624110 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7\" (UniqueName: \"kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7\") pod \"546cf8a8-7766-41dc-b7b6-a7396536b365\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " Nov 03 19:52:29.624363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.624350 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca\") pod \"546cf8a8-7766-41dc-b7b6-a7396536b365\" (UID: \"546cf8a8-7766-41dc-b7b6-a7396536b365\") " Nov 03 19:52:29.624805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.624751 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca" pod "546cf8a8-7766-41dc-b7b6-a7396536b365" (UID: "546cf8a8-7766-41dc-b7b6-a7396536b365"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:29.625688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.625163 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:29.626322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.626295 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca" pod "546cf8a8-7766-41dc-b7b6-a7396536b365" (UID: "546cf8a8-7766-41dc-b7b6-a7396536b365"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:29.626631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.626614 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7" pod "546cf8a8-7766-41dc-b7b6-a7396536b365" (UID: "546cf8a8-7766-41dc-b7b6-a7396536b365"). InnerVolumeSpecName "kube-api-access-rr7s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:29.726820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.726775 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7s7\" (UniqueName: \"kubernetes.io/projected/546cf8a8-7766-41dc-b7b6-a7396536b365-kube-api-access-rr7s7\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:29.726820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:29.726814 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546cf8a8-7766-41dc-b7b6-a7396536b365-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:30.320620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:30.320586 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" Nov 03 19:52:30.320620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:30.320607 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7" event={"ID":"546cf8a8-7766-41dc-b7b6-a7396536b365","Type":"ContainerDied","Data":"12728ad4309dc85bd3f6d11b48a7625c3785f4b2294741eba3d432637e0e697c"} Nov 03 19:52:30.320848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:30.320667 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12728ad4309dc85bd3f6d11b48a7625c3785f4b2294741eba3d432637e0e697c" Nov 03 19:52:31.032994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.028223 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:52:31.079004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.078933 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.081141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.080630 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.081141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.080701 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.081141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.080732 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.081141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.080766 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.081141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.080810 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.081870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.081841 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:31.082024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.081904 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "kube-api-access-wm7bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:31.082217 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.082194 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:31.082363 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.082232 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:31.084765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.084743 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:31.092154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.086533 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:31.092766 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.092419 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk"] Nov 03 19:52:31.097837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.097568 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6a08e-wnvkk"] Nov 03 19:52:31.181905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.181830 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.182089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.181997 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.182089 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182030 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket\") pod \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\" (UID: \"1a4b279e-a68c-4dc3-b33f-2da4afa5116d\") " Nov 03 19:52:31.182299 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182258 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:31.182384 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182319 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:31.182538 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182522 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182543 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182561 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm7bc\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-kube-api-access-wm7bc\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182577 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182592 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182607 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182623 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.182762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.182637 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.183866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.183847 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token" pod "1a4b279e-a68c-4dc3-b33f-2da4afa5116d" (UID: "1a4b279e-a68c-4dc3-b33f-2da4afa5116d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:31.294992 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.290150 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4b279e-a68c-4dc3-b33f-2da4afa5116d-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.305708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.297737 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2"] Nov 03 19:52:31.356932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.356896 5233 generic.go:356] "Generic (PLEG): container finished" podID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerID="d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5" exitCode=0 Nov 03 19:52:31.357262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.357238 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" event={"ID":"1a4b279e-a68c-4dc3-b33f-2da4afa5116d","Type":"ContainerDied","Data":"d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5"} Nov 03 19:52:31.357450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.357433 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" Nov 03 19:52:31.357635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.357617 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s" event={"ID":"1a4b279e-a68c-4dc3-b33f-2da4afa5116d","Type":"ContainerDied","Data":"9a662a5bda07d13826a01cdcbd86eaed9347729f222bd69ada1100f862e3f000"} Nov 03 19:52:31.357775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.357763 5233 scope.go:122] "RemoveContainer" containerID="d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5" Nov 03 19:52:31.400365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.400322 5233 scope.go:122] "RemoveContainer" containerID="0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b" Nov 03 19:52:31.414727 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.414698 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s"] Nov 03 19:52:31.416209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.416189 5233 scope.go:122] "RemoveContainer" containerID="d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5" Nov 03 19:52:31.416523 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:31.416504 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5\": container with ID starting with d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5 not found: ID does not exist" containerID="d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5" Nov 03 19:52:31.416633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.416533 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5"} err="failed to get container status \"d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5\": rpc error: code = NotFound desc = could not find container \"d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5\": container with ID starting with d103ae4acc3ccb5dfa00d09a6fedbab2e77bfa154ec2885239358e142ebea5e5 not found: ID does not exist" Nov 03 19:52:31.416633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.416560 5233 scope.go:122] "RemoveContainer" containerID="0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b" Nov 03 19:52:31.416827 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:31.416805 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b\": container with ID starting with 0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b not found: ID does not exist" containerID="0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b" Nov 03 19:52:31.416867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.416833 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b"} err="failed to get container status \"0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b\": rpc error: code = NotFound desc = could not find container \"0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b\": container with ID starting with 0119434b27ef5c132209947b2901e1088234b4064acdef46097869cfd0bdb15b not found: ID does not exist" Nov 03 19:52:31.421207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.421184 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b2e60-openshift-default-b47f6dd85-w4h8s"] Nov 03 19:52:31.838358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.838324 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" path="/var/lib/kubelet/pods/1a4b279e-a68c-4dc3-b33f-2da4afa5116d/volumes" Nov 03 19:52:31.838927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:31.838905 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b5da44-cf9c-4474-b1f9-c841db16285e" path="/var/lib/kubelet/pods/b0b5da44-cf9c-4474-b1f9-c841db16285e/volumes" Nov 03 19:52:32.236460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:32.236422 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z"] Nov 03 19:52:32.247638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:32.247606 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48"] Nov 03 19:52:33.887708 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.887676 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl"] Nov 03 19:52:33.888368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888350 5233 cpu_manager.go:414] "containerMap: removing container" podUID="546cf8a8-7766-41dc-b7b6-a7396536b365" containerName="revoke-keys" Nov 03 19:52:33.888368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888369 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546cf8a8-7766-41dc-b7b6-a7396536b365" containerName="revoke-keys" Nov 03 19:52:33.888466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888399 5233 cpu_manager.go:414] "containerMap: removing container" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" Nov 03 19:52:33.888466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888404 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" Nov 03 19:52:33.888466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888427 5233 cpu_manager.go:414] "containerMap: removing container" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" Nov 03 19:52:33.888466 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888434 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" Nov 03 19:52:33.888596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888518 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" Nov 03 19:52:33.888596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888530 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4b279e-a68c-4dc3-b33f-2da4afa5116d" containerName="istio-proxy" Nov 03 19:52:33.888596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.888540 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="546cf8a8-7766-41dc-b7b6-a7396536b365" containerName="revoke-keys" Nov 03 19:52:33.892160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.892138 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:33.905701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.901842 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl"] Nov 03 19:52:33.905701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.904625 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:52:33.905701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.904668 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:33.972081 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.972038 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:33.972243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.972096 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:33.972289 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:33.972248 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqbr\" (UniqueName: \"kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.073788 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.073755 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.073958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.073801 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.073958 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.073888 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqbr\" (UniqueName: \"kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.074529 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.074498 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.076316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.076292 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.086453 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.086427 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqbr\" (UniqueName: \"kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr\") pod \"maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.218141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.218112 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:34.514432 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:34.514354 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl"] Nov 03 19:52:34.520322 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:34.520288 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cab0b7_1ca2_43aa_ab55_d3ad24cc6453.slice/crio-7dfecede61b63892d13610f24c671c43000eb9c7a590784ebd7517228a9d3325 WatchSource:0}: Error finding container 7dfecede61b63892d13610f24c671c43000eb9c7a590784ebd7517228a9d3325: Status 404 returned error can't find the container with id 7dfecede61b63892d13610f24c671c43000eb9c7a590784ebd7517228a9d3325 Nov 03 19:52:35.120829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.120798 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx"] Nov 03 19:52:35.125345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.125322 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.136508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.136473 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx"] Nov 03 19:52:35.189961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.189864 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.190159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.190099 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.190159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.190151 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7mp\" (UniqueName: \"kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.291255 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.291225 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.291418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.291345 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.291418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.291408 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7mp\" (UniqueName: \"kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.292085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.292061 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.294305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.294277 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.304693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.304668 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7mp\" (UniqueName: \"kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp\") pod \"maas-api-revoke-sub-4e74dfcd-fqstx\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.428142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.428108 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" event={"ID":"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453","Type":"ContainerStarted","Data":"1b05bc7e334f5c4eaa09b4e15ac4e4074e7a108f30d8cbbd8cf694dca78a1551"} Nov 03 19:52:35.428335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.428163 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" event={"ID":"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453","Type":"ContainerStarted","Data":"7dfecede61b63892d13610f24c671c43000eb9c7a590784ebd7517228a9d3325"} Nov 03 19:52:35.448508 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.447145 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:35.455787 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.455718 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" podStartSLOduration=2.455700785 podStartE2EDuration="2.455700785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:35.448208114 +0000 UTC m=+1521.789321602" watchObservedRunningTime="2026-11-03 19:52:35.455700785 +0000 UTC m=+1521.796814272" Nov 03 19:52:35.780775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:35.780752 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx"] Nov 03 19:52:35.782795 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:35.782764 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6670ccde_86e8_4ceb_a292_40601cec7be5.slice/crio-c400a7822b2837003daacdce15f11d119fa3a53493690cfc1ff12ebdcddc5021 WatchSource:0}: Error finding container c400a7822b2837003daacdce15f11d119fa3a53493690cfc1ff12ebdcddc5021: Status 404 returned error can't find the container with id c400a7822b2837003daacdce15f11d119fa3a53493690cfc1ff12ebdcddc5021 Nov 03 19:52:36.437382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:36.437290 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" event={"ID":"6670ccde-86e8-4ceb-a292-40601cec7be5","Type":"ContainerStarted","Data":"382b63c26cee03b73e4a0b48862e1c599198ca12797edc284c0cb3a051302ab6"} Nov 03 19:52:36.437382 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:36.437348 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" event={"ID":"6670ccde-86e8-4ceb-a292-40601cec7be5","Type":"ContainerStarted","Data":"c400a7822b2837003daacdce15f11d119fa3a53493690cfc1ff12ebdcddc5021"} Nov 03 19:52:36.439280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:36.439256 5233 generic.go:356] "Generic (PLEG): container finished" podID="e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" containerID="1b05bc7e334f5c4eaa09b4e15ac4e4074e7a108f30d8cbbd8cf694dca78a1551" exitCode=0 Nov 03 19:52:36.439404 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:36.439290 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" event={"ID":"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453","Type":"ContainerDied","Data":"1b05bc7e334f5c4eaa09b4e15ac4e4074e7a108f30d8cbbd8cf694dca78a1551"} Nov 03 19:52:36.458435 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:36.458382 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" podStartSLOduration=1.458367792 podStartE2EDuration="1.458367792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:36.457380106 +0000 UTC m=+1522.798493590" watchObservedRunningTime="2026-11-03 19:52:36.458367792 +0000 UTC m=+1522.799481276" Nov 03 19:52:36.478590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:36.478562 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69"] Nov 03 19:52:37.237248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.237204 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" podUID="e6423373-89fc-451b-8b22-26e614285ad7" containerName="payload-processing" containerID="cri-o://89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08" gracePeriod=25 Nov 03 19:52:37.247952 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.247886 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerName="payload-pre-processing" containerID="cri-o://d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e" gracePeriod=25 Nov 03 19:52:37.448896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.448866 5233 generic.go:356] "Generic (PLEG): container finished" podID="6670ccde-86e8-4ceb-a292-40601cec7be5" containerID="382b63c26cee03b73e4a0b48862e1c599198ca12797edc284c0cb3a051302ab6" exitCode=0 Nov 03 19:52:37.449402 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.448956 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" event={"ID":"6670ccde-86e8-4ceb-a292-40601cec7be5","Type":"ContainerDied","Data":"382b63c26cee03b73e4a0b48862e1c599198ca12797edc284c0cb3a051302ab6"} Nov 03 19:52:37.673910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.673886 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:37.730262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.730201 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr\" (UniqueName: \"kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr\") pod \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " Nov 03 19:52:37.730262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.730266 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca\") pod \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " Nov 03 19:52:37.730625 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.730349 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca\") pod \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\" (UID: \"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453\") " Nov 03 19:52:37.731281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.731255 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca" pod "e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" (UID: "e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:37.737212 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.737185 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr" pod "e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" (UID: "e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453"). InnerVolumeSpecName "kube-api-access-7kqbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:37.739016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.738960 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca" pod "e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" (UID: "e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:37.831816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.831696 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqbr\" (UniqueName: \"kubernetes.io/projected/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-kube-api-access-7kqbr\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:37.831816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.831773 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:37.831816 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:37.831791 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:38.464700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.464675 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" Nov 03 19:52:38.465097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.464712 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl" event={"ID":"e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453","Type":"ContainerDied","Data":"7dfecede61b63892d13610f24c671c43000eb9c7a590784ebd7517228a9d3325"} Nov 03 19:52:38.465097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.464760 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfecede61b63892d13610f24c671c43000eb9c7a590784ebd7517228a9d3325" Nov 03 19:52:38.661386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.661360 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:38.751193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.751116 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca\") pod \"6670ccde-86e8-4ceb-a292-40601cec7be5\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " Nov 03 19:52:38.751193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.751185 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp\" (UniqueName: \"kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp\") pod \"6670ccde-86e8-4ceb-a292-40601cec7be5\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " Nov 03 19:52:38.751452 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.751245 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca\") pod \"6670ccde-86e8-4ceb-a292-40601cec7be5\" (UID: \"6670ccde-86e8-4ceb-a292-40601cec7be5\") " Nov 03 19:52:38.751530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.751477 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca" pod "6670ccde-86e8-4ceb-a292-40601cec7be5" (UID: "6670ccde-86e8-4ceb-a292-40601cec7be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:38.751747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.751732 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:38.753415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.753383 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp" pod "6670ccde-86e8-4ceb-a292-40601cec7be5" (UID: "6670ccde-86e8-4ceb-a292-40601cec7be5"). InnerVolumeSpecName "kube-api-access-pp7mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:38.753530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.753463 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca" pod "6670ccde-86e8-4ceb-a292-40601cec7be5" (UID: "6670ccde-86e8-4ceb-a292-40601cec7be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:38.758647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.758624 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg"] Nov 03 19:52:38.773658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.773635 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt"] Nov 03 19:52:38.853300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.853270 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7mp\" (UniqueName: \"kubernetes.io/projected/6670ccde-86e8-4ceb-a292-40601cec7be5-kube-api-access-pp7mp\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:38.853300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:38.853298 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670ccde-86e8-4ceb-a292-40601cec7be5-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.045944 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:39.045864 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j"] Nov 03 19:52:39.049794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:39.049771 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f57f0ea-w6l2j"] Nov 03 19:52:39.475556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:39.475526 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" Nov 03 19:52:39.475957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:39.475554 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx" event={"ID":"6670ccde-86e8-4ceb-a292-40601cec7be5","Type":"ContainerDied","Data":"c400a7822b2837003daacdce15f11d119fa3a53493690cfc1ff12ebdcddc5021"} Nov 03 19:52:39.475957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:39.475600 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c400a7822b2837003daacdce15f11d119fa3a53493690cfc1ff12ebdcddc5021" Nov 03 19:52:39.836919 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:39.836839 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abf34eb-8aa1-4cb1-8de9-69533130dfda" path="/var/lib/kubelet/pods/7abf34eb-8aa1-4cb1-8de9-69533130dfda/volumes" Nov 03 19:52:40.526599 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.526563 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9"] Nov 03 19:52:40.527541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.527522 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" containerName="revoke-keys" Nov 03 19:52:40.527541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.527543 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" containerName="revoke-keys" Nov 03 19:52:40.527647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.527577 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6670ccde-86e8-4ceb-a292-40601cec7be5" containerName="revoke-keys" Nov 03 19:52:40.527647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.527585 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6670ccde-86e8-4ceb-a292-40601cec7be5" containerName="revoke-keys" Nov 03 19:52:40.527725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.527713 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" containerName="revoke-keys" Nov 03 19:52:40.527764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.527730 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6670ccde-86e8-4ceb-a292-40601cec7be5" containerName="revoke-keys" Nov 03 19:52:40.532164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.532144 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.532459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.532434 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9"] Nov 03 19:52:40.537214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.537186 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.553320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.541147 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:40.553320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.541257 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:52:40.553320 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.548059 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9"] Nov 03 19:52:40.557556 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.556290 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9"] Nov 03 19:52:40.683620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.683583 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmnb\" (UniqueName: \"kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.683620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.683622 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.683798 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.683658 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.683883 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.683862 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.683927 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.683911 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.684077 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.684060 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7p5\" (UniqueName: \"kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.785877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.785793 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.785877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.785845 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.786109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.785922 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7p5\" (UniqueName: \"kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.786109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.785957 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmnb\" (UniqueName: \"kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.786109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.786016 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.786251 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.786204 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.786633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.786595 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.786759 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.786742 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.788471 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.788449 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.788602 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.788584 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.803030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.803006 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7p5\" (UniqueName: \"kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5\") pod \"maas-api-revoke-sub-584b57a5-sw6j9\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.803127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.803077 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmnb\" (UniqueName: \"kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb\") pod \"maas-api-revoke-sub-bfbcaae4-tz4r9\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:40.860537 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.856669 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:40.874051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.874022 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l"] Nov 03 19:52:40.874281 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.874252 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" containerID="cri-o://bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b" gracePeriod=30 Nov 03 19:52:40.882204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:40.882179 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:41.165588 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:41.165552 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb449d2_bb35_4efd_8607_d703c21be1b4.slice/crio-058d41a881997694af18a748435ed0e560fcb3b283db3c086637d85c9f903a32 WatchSource:0}: Error finding container 058d41a881997694af18a748435ed0e560fcb3b283db3c086637d85c9f903a32: Status 404 returned error can't find the container with id 058d41a881997694af18a748435ed0e560fcb3b283db3c086637d85c9f903a32 Nov 03 19:52:41.168274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:41.168248 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9"] Nov 03 19:52:41.394857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:41.394836 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9"] Nov 03 19:52:41.400302 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:52:41.400272 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1066d5cc_e41e_4e2d_9679_d83394ac4ffe.slice/crio-d39c94987b35c0887683f58c7045e119753a42825344031ecfdcf61e850316ea WatchSource:0}: Error finding container d39c94987b35c0887683f58c7045e119753a42825344031ecfdcf61e850316ea: Status 404 returned error can't find the container with id d39c94987b35c0887683f58c7045e119753a42825344031ecfdcf61e850316ea Nov 03 19:52:41.479063 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:41.478989 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" containerName="payload-processing" containerID="cri-o://a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5" gracePeriod=25 Nov 03 19:52:41.508823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:41.508796 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" event={"ID":"1066d5cc-e41e-4e2d-9679-d83394ac4ffe","Type":"ContainerStarted","Data":"d39c94987b35c0887683f58c7045e119753a42825344031ecfdcf61e850316ea"} Nov 03 19:52:41.510877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:41.510852 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" event={"ID":"beb449d2-bb35-4efd-8607-d703c21be1b4","Type":"ContainerStarted","Data":"058d41a881997694af18a748435ed0e560fcb3b283db3c086637d85c9f903a32"} Nov 03 19:52:41.559278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:41.559227 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" podStartSLOduration=1.559214104 podStartE2EDuration="1.559214104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:41.554089854 +0000 UTC m=+1527.895203351" watchObservedRunningTime="2026-11-03 19:52:41.559214104 +0000 UTC m=+1527.900327590" Nov 03 19:52:42.042772 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:42.042731 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" podUID="e6423373-89fc-451b-8b22-26e614285ad7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:52:42.043510 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:42.043483 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:52:42.521730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:42.521699 5233 generic.go:356] "Generic (PLEG): container finished" podID="beb449d2-bb35-4efd-8607-d703c21be1b4" containerID="49706dfc2f3d38a76458e9d2c60c4a453795b039460dfc5bfc3d5643314d096f" exitCode=0 Nov 03 19:52:42.521899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:42.521784 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" event={"ID":"beb449d2-bb35-4efd-8607-d703c21be1b4","Type":"ContainerDied","Data":"49706dfc2f3d38a76458e9d2c60c4a453795b039460dfc5bfc3d5643314d096f"} Nov 03 19:52:42.523589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:42.523558 5233 generic.go:356] "Generic (PLEG): container finished" podID="1066d5cc-e41e-4e2d-9679-d83394ac4ffe" containerID="ef17ff0410871ca31d093216b5d1398238965c24ed6f2b52922482f9fd98b409" exitCode=0 Nov 03 19:52:42.523711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:42.523592 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" event={"ID":"1066d5cc-e41e-4e2d-9679-d83394ac4ffe","Type":"ContainerDied","Data":"ef17ff0410871ca31d093216b5d1398238965c24ed6f2b52922482f9fd98b409"} Nov 03 19:52:43.743165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.743135 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:43.794645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.794579 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:43.795093 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.795071 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5\" (UniqueName: \"kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5\") pod \"beb449d2-bb35-4efd-8607-d703c21be1b4\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " Nov 03 19:52:43.795221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.795189 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca\") pod \"beb449d2-bb35-4efd-8607-d703c21be1b4\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " Nov 03 19:52:43.795336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.795316 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca\") pod \"beb449d2-bb35-4efd-8607-d703c21be1b4\" (UID: \"beb449d2-bb35-4efd-8607-d703c21be1b4\") " Nov 03 19:52:43.795750 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.795719 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca" pod "beb449d2-bb35-4efd-8607-d703c21be1b4" (UID: "beb449d2-bb35-4efd-8607-d703c21be1b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:43.795985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.795948 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:43.797882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.797855 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca" pod "beb449d2-bb35-4efd-8607-d703c21be1b4" (UID: "beb449d2-bb35-4efd-8607-d703c21be1b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:43.798051 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.797919 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5" pod "beb449d2-bb35-4efd-8607-d703c21be1b4" (UID: "beb449d2-bb35-4efd-8607-d703c21be1b4"). InnerVolumeSpecName "kube-api-access-6d7p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:43.897163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897131 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb\" (UniqueName: \"kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb\") pod \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " Nov 03 19:52:43.897321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897176 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca\") pod \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " Nov 03 19:52:43.897321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897218 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca\") pod \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\" (UID: \"1066d5cc-e41e-4e2d-9679-d83394ac4ffe\") " Nov 03 19:52:43.897603 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897575 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca" pod "1066d5cc-e41e-4e2d-9679-d83394ac4ffe" (UID: "1066d5cc-e41e-4e2d-9679-d83394ac4ffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:43.897762 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897745 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:43.897833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897762 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d7p5\" (UniqueName: \"kubernetes.io/projected/beb449d2-bb35-4efd-8607-d703c21be1b4-kube-api-access-6d7p5\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:43.897833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.897773 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb449d2-bb35-4efd-8607-d703c21be1b4-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:43.899361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.899331 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb" pod "1066d5cc-e41e-4e2d-9679-d83394ac4ffe" (UID: "1066d5cc-e41e-4e2d-9679-d83394ac4ffe"). InnerVolumeSpecName "kube-api-access-clmnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:43.899361 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.899340 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca" pod "1066d5cc-e41e-4e2d-9679-d83394ac4ffe" (UID: "1066d5cc-e41e-4e2d-9679-d83394ac4ffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:43.999563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.999526 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clmnb\" (UniqueName: \"kubernetes.io/projected/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-kube-api-access-clmnb\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:43.999563 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:43.999559 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1066d5cc-e41e-4e2d-9679-d83394ac4ffe-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:44.024054 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.024023 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp"] Nov 03 19:52:44.037877 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.037841 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n"] Nov 03 19:52:44.087751 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.087660 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:52:44.544829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.544791 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" Nov 03 19:52:44.545145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.544833 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9" event={"ID":"1066d5cc-e41e-4e2d-9679-d83394ac4ffe","Type":"ContainerDied","Data":"d39c94987b35c0887683f58c7045e119753a42825344031ecfdcf61e850316ea"} Nov 03 19:52:44.545145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.544887 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39c94987b35c0887683f58c7045e119753a42825344031ecfdcf61e850316ea" Nov 03 19:52:44.546552 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.546524 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" Nov 03 19:52:44.546683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.546577 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9" event={"ID":"beb449d2-bb35-4efd-8607-d703c21be1b4","Type":"ContainerDied","Data":"058d41a881997694af18a748435ed0e560fcb3b283db3c086637d85c9f903a32"} Nov 03 19:52:44.546683 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:44.546616 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058d41a881997694af18a748435ed0e560fcb3b283db3c086637d85c9f903a32" Nov 03 19:52:45.633996 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.633156 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz"] Nov 03 19:52:45.634509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.634155 5233 cpu_manager.go:414] "containerMap: removing container" podUID="beb449d2-bb35-4efd-8607-d703c21be1b4" containerName="revoke-keys" Nov 03 19:52:45.634509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.634176 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb449d2-bb35-4efd-8607-d703c21be1b4" containerName="revoke-keys" Nov 03 19:52:45.634509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.634206 5233 cpu_manager.go:414] "containerMap: removing container" podUID="1066d5cc-e41e-4e2d-9679-d83394ac4ffe" containerName="revoke-keys" Nov 03 19:52:45.634509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.634215 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1066d5cc-e41e-4e2d-9679-d83394ac4ffe" containerName="revoke-keys" Nov 03 19:52:45.634509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.634393 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb449d2-bb35-4efd-8607-d703c21be1b4" containerName="revoke-keys" Nov 03 19:52:45.634509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.634410 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="1066d5cc-e41e-4e2d-9679-d83394ac4ffe" containerName="revoke-keys" Nov 03 19:52:45.652742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.651620 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz"] Nov 03 19:52:45.652742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.651733 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.658368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.658301 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:52:45.658368 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.658299 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:45.756847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.756814 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvj5g\" (UniqueName: \"kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.757052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.756875 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.757052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.756963 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.858356 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.858326 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvj5g\" (UniqueName: \"kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.858564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.858373 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.858564 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.858460 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.859268 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.859239 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.861030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.861004 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.871554 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.871534 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvj5g\" (UniqueName: \"kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g\") pod \"maas-api-revoke-sub-4c393a05-2vxgz\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:45.972530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:45.972494 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:46.224180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.224144 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:52:46.318941 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.318907 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz"] Nov 03 19:52:46.365487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365465 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.365620 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365606 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.365672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365662 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.365719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365701 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.365878 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365852 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.366018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365904 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.366018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.365952 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.366145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366022 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.366145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366030 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.366145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366047 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.366145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366077 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket\") pod \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\" (UID: \"5f33cd4c-323e-4fe9-9820-a5e1b9342431\") " Nov 03 19:52:46.366408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366180 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.366408 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366219 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:46.366521 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366464 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.366722 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366698 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.366777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366724 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.366777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366739 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.366777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366754 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.366777 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.366768 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.368049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.368022 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:46.368145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.368092 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "kube-api-access-qcxr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:46.368197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.368143 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.368378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.368362 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token" pod "5f33cd4c-323e-4fe9-9820-a5e1b9342431" (UID: "5f33cd4c-323e-4fe9-9820-a5e1b9342431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:46.387044 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.387012 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" containerID="cri-o://6297e818c7a47e8d29fc99f24ad5aec9d33042f4291732483b153a24ec88ebc7" gracePeriod=45 Nov 03 19:52:46.468148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.468122 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcxr6\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-kube-api-access-qcxr6\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.468148 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.468150 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.468308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.468159 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.468308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.468169 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f33cd4c-323e-4fe9-9820-a5e1b9342431-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.574280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.574242 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" event={"ID":"8c8aece6-6558-4335-9410-18cfe23ec8b7","Type":"ContainerStarted","Data":"3eb87a7e513836631095683f4837fcbc9507afe79e4840a8c02c315e5a02601b"} Nov 03 19:52:46.576700 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.576669 5233 generic.go:356] "Generic (PLEG): container finished" podID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerID="6297e818c7a47e8d29fc99f24ad5aec9d33042f4291732483b153a24ec88ebc7" exitCode=0 Nov 03 19:52:46.576841 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.576771 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" event={"ID":"86065665-60a4-4bce-89b6-3979cfe52fe9","Type":"ContainerDied","Data":"6297e818c7a47e8d29fc99f24ad5aec9d33042f4291732483b153a24ec88ebc7"} Nov 03 19:52:46.578861 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.578838 5233 generic.go:356] "Generic (PLEG): container finished" podID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerID="bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b" exitCode=0 Nov 03 19:52:46.579013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.578996 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" event={"ID":"5f33cd4c-323e-4fe9-9820-a5e1b9342431","Type":"ContainerDied","Data":"bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b"} Nov 03 19:52:46.579085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.579039 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" event={"ID":"5f33cd4c-323e-4fe9-9820-a5e1b9342431","Type":"ContainerDied","Data":"2444a5a45fd320bc6f436f9487e002c4770bc3a9f0270502e62b8b5ddb1d219f"} Nov 03 19:52:46.579085 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.579071 5233 scope.go:122] "RemoveContainer" containerID="bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b" Nov 03 19:52:46.579241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.579228 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l" Nov 03 19:52:46.605736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.605257 5233 scope.go:122] "RemoveContainer" containerID="bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b" Nov 03 19:52:46.605736 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:46.605576 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b\": container with ID starting with bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b not found: ID does not exist" containerID="bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b" Nov 03 19:52:46.605736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.605610 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b"} err="failed to get container status \"bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b\": rpc error: code = NotFound desc = could not find container \"bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b\": container with ID starting with bce2aa960f5990165bf5090510b4f5371d3e808d9a50b0c6cda45c2b3a7da25b not found: ID does not exist" Nov 03 19:52:46.647415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.647373 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l"] Nov 03 19:52:46.663717 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.663683 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:52:46.664309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.664284 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c6a08e-openshift-default-7945f786df-zg44l"] Nov 03 19:52:46.772532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.772431 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir\") pod \"86065665-60a4-4bce-89b6-3979cfe52fe9\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " Nov 03 19:52:46.772532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.772529 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm\") pod \"86065665-60a4-4bce-89b6-3979cfe52fe9\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " Nov 03 19:52:46.772744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.772693 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home\") pod \"86065665-60a4-4bce-89b6-3979cfe52fe9\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " Nov 03 19:52:46.772744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.772735 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h\" (UniqueName: \"kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h\") pod \"86065665-60a4-4bce-89b6-3979cfe52fe9\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " Nov 03 19:52:46.772820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.772791 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache\") pod \"86065665-60a4-4bce-89b6-3979cfe52fe9\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " Nov 03 19:52:46.772902 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.772886 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs\" (UniqueName: \"kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs\") pod \"86065665-60a4-4bce-89b6-3979cfe52fe9\" (UID: \"86065665-60a4-4bce-89b6-3979cfe52fe9\") " Nov 03 19:52:46.773416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.773388 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir" pod "86065665-60a4-4bce-89b6-3979cfe52fe9" (UID: "86065665-60a4-4bce-89b6-3979cfe52fe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.774332 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.774159 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache" pod "86065665-60a4-4bce-89b6-3979cfe52fe9" (UID: "86065665-60a4-4bce-89b6-3979cfe52fe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.774525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.774487 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home" pod "86065665-60a4-4bce-89b6-3979cfe52fe9" (UID: "86065665-60a4-4bce-89b6-3979cfe52fe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.775469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.775448 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.775592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.775580 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.775689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.775677 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.781447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.780229 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h" pod "86065665-60a4-4bce-89b6-3979cfe52fe9" (UID: "86065665-60a4-4bce-89b6-3979cfe52fe9"). InnerVolumeSpecName "kube-api-access-dlg8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:46.781447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.780258 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm" pod "86065665-60a4-4bce-89b6-3979cfe52fe9" (UID: "86065665-60a4-4bce-89b6-3979cfe52fe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:46.781447 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.780301 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs" pod "86065665-60a4-4bce-89b6-3979cfe52fe9" (UID: "86065665-60a4-4bce-89b6-3979cfe52fe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:46.877817 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.877783 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86065665-60a4-4bce-89b6-3979cfe52fe9-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.878033 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.878013 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlg8h\" (UniqueName: \"kubernetes.io/projected/86065665-60a4-4bce-89b6-3979cfe52fe9-kube-api-access-dlg8h\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:46.878145 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:46.878130 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86065665-60a4-4bce-89b6-3979cfe52fe9-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:47.597418 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.597374 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" event={"ID":"86065665-60a4-4bce-89b6-3979cfe52fe9","Type":"ContainerDied","Data":"f8d4b1044c1ea6650892dbc287571777bea9ab3608ae55cfcf2d167494525572"} Nov 03 19:52:47.597627 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.597442 5233 scope.go:122] "RemoveContainer" containerID="6297e818c7a47e8d29fc99f24ad5aec9d33042f4291732483b153a24ec88ebc7" Nov 03 19:52:47.597687 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.597638 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2" Nov 03 19:52:47.605027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.604882 5233 generic.go:356] "Generic (PLEG): container finished" podID="8c8aece6-6558-4335-9410-18cfe23ec8b7" containerID="ab19f8a3d109db66d234eaf68fbe046d0e7c703ba8c99d9cdfbe7b47aef803a1" exitCode=0 Nov 03 19:52:47.605168 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.605144 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" event={"ID":"8c8aece6-6558-4335-9410-18cfe23ec8b7","Type":"ContainerDied","Data":"ab19f8a3d109db66d234eaf68fbe046d0e7c703ba8c99d9cdfbe7b47aef803a1"} Nov 03 19:52:47.643569 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.643539 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2"] Nov 03 19:52:47.648482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.648451 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-036184/distinct-w3-036184-kserve-c7c86cf5d-677h2"] Nov 03 19:52:47.839704 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.839668 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" path="/var/lib/kubelet/pods/5f33cd4c-323e-4fe9-9820-a5e1b9342431/volumes" Nov 03 19:52:47.840176 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:47.840158 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" path="/var/lib/kubelet/pods/86065665-60a4-4bce-89b6-3979cfe52fe9/volumes" Nov 03 19:52:48.812503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.812469 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:48.913931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.913843 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca\") pod \"8c8aece6-6558-4335-9410-18cfe23ec8b7\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " Nov 03 19:52:48.914129 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.914037 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca\") pod \"8c8aece6-6558-4335-9410-18cfe23ec8b7\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " Nov 03 19:52:48.914129 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.914078 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g\" (UniqueName: \"kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g\") pod \"8c8aece6-6558-4335-9410-18cfe23ec8b7\" (UID: \"8c8aece6-6558-4335-9410-18cfe23ec8b7\") " Nov 03 19:52:48.914641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.914615 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca" pod "8c8aece6-6558-4335-9410-18cfe23ec8b7" (UID: "8c8aece6-6558-4335-9410-18cfe23ec8b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:48.916701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.916677 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g" pod "8c8aece6-6558-4335-9410-18cfe23ec8b7" (UID: "8c8aece6-6558-4335-9410-18cfe23ec8b7"). InnerVolumeSpecName "kube-api-access-pvj5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:48.916997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:48.916946 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca" pod "8c8aece6-6558-4335-9410-18cfe23ec8b7" (UID: "8c8aece6-6558-4335-9410-18cfe23ec8b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:49.019782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.019736 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.019782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.019770 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aece6-6558-4335-9410-18cfe23ec8b7-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.019782 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.019780 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvj5g\" (UniqueName: \"kubernetes.io/projected/8c8aece6-6558-4335-9410-18cfe23ec8b7-kube-api-access-pvj5g\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.025321 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.025288 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" podUID="610b32be-07bd-4b82-89cb-ad4b63638981" containerName="payload-processing" containerID="cri-o://6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f" gracePeriod=25 Nov 03 19:52:49.042764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.042731 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" podUID="ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" containerName="payload-pre-processing" containerID="cri-o://4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7" gracePeriod=25 Nov 03 19:52:49.335651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.335620 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:52:49.410132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.410108 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:52:49.426184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426156 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume\") pod \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " Nov 03 19:52:49.426184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426193 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m\" (UniqueName: \"kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m\") pod \"610b32be-07bd-4b82-89cb-ad4b63638981\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " Nov 03 19:52:49.426416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426272 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume\") pod \"610b32be-07bd-4b82-89cb-ad4b63638981\" (UID: \"610b32be-07bd-4b82-89cb-ad4b63638981\") " Nov 03 19:52:49.426479 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426446 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w\" (UniqueName: \"kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w\") pod \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\" (UID: \"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5\") " Nov 03 19:52:49.426621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426592 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume" pod "ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" (UID: "ee48163e-ed96-4d75-8cc9-a86c1c0e35c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:49.426898 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426871 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume" pod "610b32be-07bd-4b82-89cb-ad4b63638981" (UID: "610b32be-07bd-4b82-89cb-ad4b63638981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:49.426993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.426884 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.429004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.428911 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w" pod "ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" (UID: "ee48163e-ed96-4d75-8cc9-a86c1c0e35c5"). InnerVolumeSpecName "kube-api-access-8qr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:49.429004 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.428932 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m" pod "610b32be-07bd-4b82-89cb-ad4b63638981" (UID: "610b32be-07bd-4b82-89cb-ad4b63638981"). InnerVolumeSpecName "kube-api-access-bwh9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:49.528328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.528238 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwh9m\" (UniqueName: \"kubernetes.io/projected/610b32be-07bd-4b82-89cb-ad4b63638981-kube-api-access-bwh9m\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.528328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.528266 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610b32be-07bd-4b82-89cb-ad4b63638981-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.528328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.528277 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qr6w\" (UniqueName: \"kubernetes.io/projected/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5-kube-api-access-8qr6w\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:49.627674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.627636 5233 generic.go:356] "Generic (PLEG): container finished" podID="ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" containerID="4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7" exitCode=0 Nov 03 19:52:49.627824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.627683 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" Nov 03 19:52:49.627824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.627715 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" event={"ID":"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5","Type":"ContainerDied","Data":"4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7"} Nov 03 19:52:49.627824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.627767 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n" event={"ID":"ee48163e-ed96-4d75-8cc9-a86c1c0e35c5","Type":"ContainerDied","Data":"753322f563a3249abbdd82edb9e86f1a426842b9ecbe75a0e7f63d59a816697d"} Nov 03 19:52:49.627824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.627788 5233 scope.go:122] "RemoveContainer" containerID="4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7" Nov 03 19:52:49.629870 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.629839 5233 generic.go:356] "Generic (PLEG): container finished" podID="610b32be-07bd-4b82-89cb-ad4b63638981" containerID="6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f" exitCode=0 Nov 03 19:52:49.630005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.629906 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" Nov 03 19:52:49.630005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.629943 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" event={"ID":"610b32be-07bd-4b82-89cb-ad4b63638981","Type":"ContainerDied","Data":"6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f"} Nov 03 19:52:49.630140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.630015 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp" event={"ID":"610b32be-07bd-4b82-89cb-ad4b63638981","Type":"ContainerDied","Data":"3974733b717e333b8bfb4780766a50790db62b3df5f92a49f3568e558ab9b4f7"} Nov 03 19:52:49.636343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.636319 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" event={"ID":"8c8aece6-6558-4335-9410-18cfe23ec8b7","Type":"ContainerDied","Data":"3eb87a7e513836631095683f4837fcbc9507afe79e4840a8c02c315e5a02601b"} Nov 03 19:52:49.636343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.636333 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz" Nov 03 19:52:49.636535 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.636361 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb87a7e513836631095683f4837fcbc9507afe79e4840a8c02c315e5a02601b" Nov 03 19:52:49.645730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.645710 5233 scope.go:122] "RemoveContainer" containerID="4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7" Nov 03 19:52:49.646113 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:49.646090 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7\": container with ID starting with 4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7 not found: ID does not exist" containerID="4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7" Nov 03 19:52:49.646224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.646127 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7"} err="failed to get container status \"4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7\": rpc error: code = NotFound desc = could not find container \"4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7\": container with ID starting with 4aa1454062f0a7d24184c2e0c476b960f6f7bb894f35b2e225fd69ebceeba9d7 not found: ID does not exist" Nov 03 19:52:49.646224 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.646151 5233 scope.go:122] "RemoveContainer" containerID="6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f" Nov 03 19:52:49.661954 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.661913 5233 scope.go:122] "RemoveContainer" containerID="6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f" Nov 03 19:52:49.662224 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:49.662205 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f\": container with ID starting with 6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f not found: ID does not exist" containerID="6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f" Nov 03 19:52:49.662280 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.662231 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f"} err="failed to get container status \"6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f\": rpc error: code = NotFound desc = could not find container \"6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f\": container with ID starting with 6ee51689aa3b0c095303d0377233ac88727ceb98e027a925d5b3bbaa762eae8f not found: ID does not exist" Nov 03 19:52:49.672330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.672309 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n"] Nov 03 19:52:49.679065 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.679037 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-435bd8-7cb5df5ff-4lh2n"] Nov 03 19:52:49.696514 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.696356 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp"] Nov 03 19:52:49.702425 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.702399 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-435bd8-8968bdbc-wmcqp"] Nov 03 19:52:49.841398 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.841364 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610b32be-07bd-4b82-89cb-ad4b63638981" path="/var/lib/kubelet/pods/610b32be-07bd-4b82-89cb-ad4b63638981/volumes" Nov 03 19:52:49.851530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:49.851497 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" path="/var/lib/kubelet/pods/ee48163e-ed96-4d75-8cc9-a86c1c0e35c5/volumes" Nov 03 19:52:51.279331 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:51.279282 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:52:51.656530 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:51.656362 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp"] Nov 03 19:52:52.043137 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:52.043099 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" podUID="e6423373-89fc-451b-8b22-26e614285ad7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:52:52.043316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:52.043189 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:52:52.044899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:52.044878 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm"] Nov 03 19:52:52.052046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:52.052017 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220955f5-4mxtm"] Nov 03 19:52:53.783375 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:53.783336 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" containerID="cri-o://8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18" gracePeriod=45 Nov 03 19:52:53.815282 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:53.814545 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" containerID="cri-o://f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99" gracePeriod=45 Nov 03 19:52:53.847930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:53.847895 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef18dce2-81af-4133-972c-183253833e16" path="/var/lib/kubelet/pods/ef18dce2-81af-4133-972c-183253833e16/volumes" Nov 03 19:52:54.110165 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.109323 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:52:54.115258 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.115229 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h"] Nov 03 19:52:54.122867 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.122840 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-05e217-dbp8h"] Nov 03 19:52:54.138191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.138150 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 19:52:54.220027 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.219953 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:52:54.276141 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.276115 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:52:54.342045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.339079 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home\") pod \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " Nov 03 19:52:54.342501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.342477 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm\") pod \"319cda80-14b5-4529-8cc6-298f6b2e0353\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " Nov 03 19:52:54.342686 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.342669 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home\") pod \"319cda80-14b5-4529-8cc6-298f6b2e0353\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " Nov 03 19:52:54.345193 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.342857 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h\" (UniqueName: \"kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h\") pod \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " Nov 03 19:52:54.345384 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.345362 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache\") pod \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " Nov 03 19:52:54.345614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.345595 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache\") pod \"319cda80-14b5-4529-8cc6-298f6b2e0353\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " Nov 03 19:52:54.345746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.345731 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir\") pod \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " Nov 03 19:52:54.345886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.345873 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x\" (UniqueName: \"kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x\") pod \"319cda80-14b5-4529-8cc6-298f6b2e0353\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " Nov 03 19:52:54.346045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.346031 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs\" (UniqueName: \"kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs\") pod \"319cda80-14b5-4529-8cc6-298f6b2e0353\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " Nov 03 19:52:54.346234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.346222 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir\") pod \"319cda80-14b5-4529-8cc6-298f6b2e0353\" (UID: \"319cda80-14b5-4529-8cc6-298f6b2e0353\") " Nov 03 19:52:54.346523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.346498 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm\") pod \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " Nov 03 19:52:54.347829 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.347808 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs\") pod \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\" (UID: \"d98abe45-77ff-413a-8ea8-c88f9a7a26cd\") " Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.341223 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home" pod "d98abe45-77ff-413a-8ea8-c88f9a7a26cd" (UID: "d98abe45-77ff-413a-8ea8-c88f9a7a26cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.344428 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home" pod "319cda80-14b5-4529-8cc6-298f6b2e0353" (UID: "319cda80-14b5-4529-8cc6-298f6b2e0353"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.346747 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir" pod "319cda80-14b5-4529-8cc6-298f6b2e0353" (UID: "319cda80-14b5-4529-8cc6-298f6b2e0353"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.347321 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache" pod "d98abe45-77ff-413a-8ea8-c88f9a7a26cd" (UID: "d98abe45-77ff-413a-8ea8-c88f9a7a26cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.347348 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir" pod "d98abe45-77ff-413a-8ea8-c88f9a7a26cd" (UID: "d98abe45-77ff-413a-8ea8-c88f9a7a26cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.347138 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache" pod "319cda80-14b5-4529-8cc6-298f6b2e0353" (UID: "319cda80-14b5-4529-8cc6-298f6b2e0353"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.347420 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h" pod "d98abe45-77ff-413a-8ea8-c88f9a7a26cd" (UID: "d98abe45-77ff-413a-8ea8-c88f9a7a26cd"). InnerVolumeSpecName "kube-api-access-8qd2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:54.349394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.349341 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x" pod "319cda80-14b5-4529-8cc6-298f6b2e0353" (UID: "319cda80-14b5-4529-8cc6-298f6b2e0353"). InnerVolumeSpecName "kube-api-access-gxq9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:54.349896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.349657 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm" pod "d98abe45-77ff-413a-8ea8-c88f9a7a26cd" (UID: "d98abe45-77ff-413a-8ea8-c88f9a7a26cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.349896 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.349668 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs" pod "319cda80-14b5-4529-8cc6-298f6b2e0353" (UID: "319cda80-14b5-4529-8cc6-298f6b2e0353"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:54.350202 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.350178 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm" pod "319cda80-14b5-4529-8cc6-298f6b2e0353" (UID: "319cda80-14b5-4529-8cc6-298f6b2e0353"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:54.354152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.354121 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs" pod "d98abe45-77ff-413a-8ea8-c88f9a7a26cd" (UID: "d98abe45-77ff-413a-8ea8-c88f9a7a26cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:54.450783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450745 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.450783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450780 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.450783 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450789 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450799 5233 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-home\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450818 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qd2h\" (UniqueName: \"kubernetes.io/projected/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-kube-api-access-8qd2h\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450828 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450837 5233 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-model-cache\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450846 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450854 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxq9x\" (UniqueName: \"kubernetes.io/projected/319cda80-14b5-4529-8cc6-298f6b2e0353-kube-api-access-gxq9x\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450862 5233 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319cda80-14b5-4529-8cc6-298f6b2e0353-tls-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450870 5233 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/319cda80-14b5-4529-8cc6-298f6b2e0353-tmp-dir\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.451096 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.450877 5233 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98abe45-77ff-413a-8ea8-c88f9a7a26cd-dshm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:54.649171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.647032 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7"] Nov 03 19:52:54.677112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.676651 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-036184-2fbh7"] Nov 03 19:52:54.740993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.740680 5233 generic.go:356] "Generic (PLEG): container finished" podID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerID="8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18" exitCode=0 Nov 03 19:52:54.740993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.740771 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" event={"ID":"319cda80-14b5-4529-8cc6-298f6b2e0353","Type":"ContainerDied","Data":"8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18"} Nov 03 19:52:54.740993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.740807 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" event={"ID":"319cda80-14b5-4529-8cc6-298f6b2e0353","Type":"ContainerDied","Data":"b5f0a9fbcc00d4ee3601d3ffe44b9af456bb31c3a01a6936d7b937ef4bbb7730"} Nov 03 19:52:54.740993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.740831 5233 scope.go:122] "RemoveContainer" containerID="8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18" Nov 03 19:52:54.741352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.741026 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" Nov 03 19:52:54.771957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.771916 5233 generic.go:356] "Generic (PLEG): container finished" podID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerID="f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99" exitCode=0 Nov 03 19:52:54.772123 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.772013 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" event={"ID":"d98abe45-77ff-413a-8ea8-c88f9a7a26cd","Type":"ContainerDied","Data":"f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99"} Nov 03 19:52:54.772309 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.772289 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" Nov 03 19:52:54.786367 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.785050 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt" event={"ID":"d98abe45-77ff-413a-8ea8-c88f9a7a26cd","Type":"ContainerDied","Data":"e7056565330b2cccb2f16544de50b8bfc9b1c7e015d2f95b85184bdb73f6829c"} Nov 03 19:52:54.927590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.918143 5233 scope.go:122] "RemoveContainer" containerID="8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18" Nov 03 19:52:54.927590 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:54.918564 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18\": container with ID starting with 8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18 not found: ID does not exist" containerID="8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18" Nov 03 19:52:54.927590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.918606 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18"} err="failed to get container status \"8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18\": rpc error: code = NotFound desc = could not find container \"8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18\": container with ID starting with 8594edc78a1dc70b4ca3da7e212be329b37a1076e68df4bfb6865bafab6dbe18 not found: ID does not exist" Nov 03 19:52:54.927590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.918633 5233 scope.go:122] "RemoveContainer" containerID="f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99" Nov 03 19:52:54.956163 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.956135 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg"] Nov 03 19:52:54.976730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.976698 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg"] Nov 03 19:52:55.003010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:54.998142 5233 scope.go:122] "RemoveContainer" containerID="f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99" Nov 03 19:52:55.037040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.029036 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt"] Nov 03 19:52:55.037040 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:55.029564 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99\": container with ID starting with f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99 not found: ID does not exist" containerID="f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99" Nov 03 19:52:55.037040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.029613 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99"} err="failed to get container status \"f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99\": rpc error: code = NotFound desc = could not find container \"f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99\": container with ID starting with f6615750f86ea1635171b71687309db8d4511dceb7e9c922bd6c5e61e1f56d99 not found: ID does not exist" Nov 03 19:52:55.043487 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.043461 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-435bd8/premium-w2-435bd8-kserve-697d5d5c46-478tt"] Nov 03 19:52:55.081001 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.076051 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-435bd8/distinct-w2-435bd8-kserve-756fd644fc-p6zqg" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 19:52:55.838341 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.838237 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" path="/var/lib/kubelet/pods/319cda80-14b5-4529-8cc6-298f6b2e0353/volumes" Nov 03 19:52:55.838882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.838661 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546cf8a8-7766-41dc-b7b6-a7396536b365" path="/var/lib/kubelet/pods/546cf8a8-7766-41dc-b7b6-a7396536b365/volumes" Nov 03 19:52:55.839088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.839072 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc527f6-c48e-42ac-8583-2965cb822685" path="/var/lib/kubelet/pods/9cc527f6-c48e-42ac-8583-2965cb822685/volumes" Nov 03 19:52:55.839734 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:55.839709 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" path="/var/lib/kubelet/pods/d98abe45-77ff-413a-8ea8-c88f9a7a26cd/volumes" Nov 03 19:52:56.663287 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:56.663156 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" podUID="a8705ab5-c573-4712-97f9-4bc7ec7d97fc" containerName="payload-pre-processing" containerID="cri-o://48dc654c47f2a21b731214a25cbf139c9b1aea9b8e99fc4e068437466cdb3bac" gracePeriod=25 Nov 03 19:52:56.730345 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:56.730309 5233 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8705ab5_c573_4712_97f9_4bc7ec7d97fc.slice/crio-conmon-48dc654c47f2a21b731214a25cbf139c9b1aea9b8e99fc4e068437466cdb3bac.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:52:56.812142 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:56.812104 5233 generic.go:356] "Generic (PLEG): container finished" podID="a8705ab5-c573-4712-97f9-4bc7ec7d97fc" containerID="48dc654c47f2a21b731214a25cbf139c9b1aea9b8e99fc4e068437466cdb3bac" exitCode=0 Nov 03 19:52:56.812302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:56.812146 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" event={"ID":"a8705ab5-c573-4712-97f9-4bc7ec7d97fc","Type":"ContainerDied","Data":"48dc654c47f2a21b731214a25cbf139c9b1aea9b8e99fc4e068437466cdb3bac"} Nov 03 19:52:57.039352 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.038995 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:52:57.202800 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.202693 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume\") pod \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " Nov 03 19:52:57.203016 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.202946 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w\" (UniqueName: \"kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w\") pod \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\" (UID: \"a8705ab5-c573-4712-97f9-4bc7ec7d97fc\") " Nov 03 19:52:57.203178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.203154 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume" pod "a8705ab5-c573-4712-97f9-4bc7ec7d97fc" (UID: "a8705ab5-c573-4712-97f9-4bc7ec7d97fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:57.203544 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.203512 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.205191 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.205164 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w" pod "a8705ab5-c573-4712-97f9-4bc7ec7d97fc" (UID: "a8705ab5-c573-4712-97f9-4bc7ec7d97fc"). InnerVolumeSpecName "kube-api-access-9ch6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:57.224784 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.224750 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl"] Nov 03 19:52:57.228587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.228563 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-435bd8-jlqpl"] Nov 03 19:52:57.304395 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.304365 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ch6w\" (UniqueName: \"kubernetes.io/projected/a8705ab5-c573-4712-97f9-4bc7ec7d97fc-kube-api-access-9ch6w\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.569509 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.569477 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:52:57.632507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.632465 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:52:57.722412 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.722335 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume\") pod \"e6423373-89fc-451b-8b22-26e614285ad7\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " Nov 03 19:52:57.722647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.722629 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq\" (UniqueName: \"kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq\") pod \"fb5540ea-5820-49ef-918f-655bcffeba0c\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " Nov 03 19:52:57.722887 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.722870 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume\") pod \"fb5540ea-5820-49ef-918f-655bcffeba0c\" (UID: \"fb5540ea-5820-49ef-918f-655bcffeba0c\") " Nov 03 19:52:57.723114 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.723097 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct\" (UniqueName: \"kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct\") pod \"e6423373-89fc-451b-8b22-26e614285ad7\" (UID: \"e6423373-89fc-451b-8b22-26e614285ad7\") " Nov 03 19:52:57.724596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.724555 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume" pod "fb5540ea-5820-49ef-918f-655bcffeba0c" (UID: "fb5540ea-5820-49ef-918f-655bcffeba0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:57.724596 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.724578 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume" pod "e6423373-89fc-451b-8b22-26e614285ad7" (UID: "e6423373-89fc-451b-8b22-26e614285ad7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:57.727534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.727487 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq" pod "fb5540ea-5820-49ef-918f-655bcffeba0c" (UID: "fb5540ea-5820-49ef-918f-655bcffeba0c"). InnerVolumeSpecName "kube-api-access-rdgtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:57.735158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.735130 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct" pod "e6423373-89fc-451b-8b22-26e614285ad7" (UID: "e6423373-89fc-451b-8b22-26e614285ad7"). InnerVolumeSpecName "kube-api-access-85hct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:57.854464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.833426 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdgtq\" (UniqueName: \"kubernetes.io/projected/fb5540ea-5820-49ef-918f-655bcffeba0c-kube-api-access-rdgtq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.854464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.833470 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5540ea-5820-49ef-918f-655bcffeba0c-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.854464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.833486 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hct\" (UniqueName: \"kubernetes.io/projected/e6423373-89fc-451b-8b22-26e614285ad7-kube-api-access-85hct\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.854464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.833499 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6423373-89fc-451b-8b22-26e614285ad7-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:57.897961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.895338 5233 generic.go:356] "Generic (PLEG): container finished" podID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerID="d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e" exitCode=0 Nov 03 19:52:57.897961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.895513 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" Nov 03 19:52:57.903048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.902945 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" Nov 03 19:52:57.906210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.905210 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453" path="/var/lib/kubelet/pods/e8cab0b7-1ca2-43aa-ab55-d3ad24cc6453/volumes" Nov 03 19:52:57.906210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.905833 5233 generic.go:356] "Generic (PLEG): container finished" podID="e6423373-89fc-451b-8b22-26e614285ad7" containerID="89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08" exitCode=0 Nov 03 19:52:57.906210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.905993 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" Nov 03 19:52:57.906210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.906136 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" event={"ID":"fb5540ea-5820-49ef-918f-655bcffeba0c","Type":"ContainerDied","Data":"d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e"} Nov 03 19:52:57.907200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.906187 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48" event={"ID":"fb5540ea-5820-49ef-918f-655bcffeba0c","Type":"ContainerDied","Data":"aca6286b0e1e188afa1a8886eac0ed70c325adc5f0f3d69348d99d7f2b12128e"} Nov 03 19:52:57.907200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.906575 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp" event={"ID":"a8705ab5-c573-4712-97f9-4bc7ec7d97fc","Type":"ContainerDied","Data":"fd59dd4270654c19ceaad55d9077a47db1fb61e332153f47e651ee76fde25806"} Nov 03 19:52:57.907200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.906603 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" event={"ID":"e6423373-89fc-451b-8b22-26e614285ad7","Type":"ContainerDied","Data":"89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08"} Nov 03 19:52:57.907200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.906627 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z" event={"ID":"e6423373-89fc-451b-8b22-26e614285ad7","Type":"ContainerDied","Data":"94cf5eee04803a6ad519c0e0ce8690e50366c6d97a7db5007008f713038a0a28"} Nov 03 19:52:57.907200 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.906656 5233 scope.go:122] "RemoveContainer" containerID="d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e" Nov 03 19:52:57.933825 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.933800 5233 scope.go:122] "RemoveContainer" containerID="d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e" Nov 03 19:52:57.935523 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:57.934945 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e\": container with ID starting with d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e not found: ID does not exist" containerID="d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e" Nov 03 19:52:57.935523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.934999 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e"} err="failed to get container status \"d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e\": rpc error: code = NotFound desc = could not find container \"d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e\": container with ID starting with d7eb16af918a8c75eb873546457185cea1b440464ce495c446e0fedb84b9208e not found: ID does not exist" Nov 03 19:52:57.935523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.935038 5233 scope.go:122] "RemoveContainer" containerID="48dc654c47f2a21b731214a25cbf139c9b1aea9b8e99fc4e068437466cdb3bac" Nov 03 19:52:57.948581 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.948546 5233 scope.go:122] "RemoveContainer" containerID="89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08" Nov 03 19:52:57.957269 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.957227 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z"] Nov 03 19:52:57.978793 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.965562 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-05e217-85dcd6bf97-ccz5z"] Nov 03 19:52:57.983465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.983440 5233 scope.go:122] "RemoveContainer" containerID="89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08" Nov 03 19:52:57.987952 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:52:57.987920 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08\": container with ID starting with 89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08 not found: ID does not exist" containerID="89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08" Nov 03 19:52:57.988152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.988120 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08"} err="failed to get container status \"89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08\": rpc error: code = NotFound desc = could not find container \"89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08\": container with ID starting with 89c3c2920503488141c4d6f374fcc2dcf4ac58a796949f1e655d377d0b854c08 not found: ID does not exist" Nov 03 19:52:57.994738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:57.994716 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp"] Nov 03 19:52:58.000765 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:58.000733 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-763a69-5d8fcbc4c-sl8cp"] Nov 03 19:52:58.020984 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:58.020943 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48"] Nov 03 19:52:58.026932 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:58.026906 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-05e217-557769678c-djc48"] Nov 03 19:52:59.903235 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:59.903194 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8705ab5-c573-4712-97f9-4bc7ec7d97fc" path="/var/lib/kubelet/pods/a8705ab5-c573-4712-97f9-4bc7ec7d97fc/volumes" Nov 03 19:52:59.904845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:59.904817 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6423373-89fc-451b-8b22-26e614285ad7" path="/var/lib/kubelet/pods/e6423373-89fc-451b-8b22-26e614285ad7/volumes" Nov 03 19:52:59.905769 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:52:59.905751 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" path="/var/lib/kubelet/pods/fb5540ea-5820-49ef-918f-655bcffeba0c/volumes" Nov 03 19:53:01.761190 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.761125 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:53:01.795770 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.795727 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9\" (UniqueName: \"kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9\") pod \"9359b24e-557f-428e-8205-5c9035c3f76e\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " Nov 03 19:53:01.796076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.796052 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume\") pod \"9359b24e-557f-428e-8205-5c9035c3f76e\" (UID: \"9359b24e-557f-428e-8205-5c9035c3f76e\") " Nov 03 19:53:01.796516 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.796487 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume" pod "9359b24e-557f-428e-8205-5c9035c3f76e" (UID: "9359b24e-557f-428e-8205-5c9035c3f76e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:01.798695 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.798655 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9" pod "9359b24e-557f-428e-8205-5c9035c3f76e" (UID: "9359b24e-557f-428e-8205-5c9035c3f76e"). InnerVolumeSpecName "kube-api-access-w96k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:01.928223 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.928179 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9359b24e-557f-428e-8205-5c9035c3f76e-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:01.928223 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.928218 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96k9\" (UniqueName: \"kubernetes.io/projected/9359b24e-557f-428e-8205-5c9035c3f76e-kube-api-access-w96k9\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:01.977378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.977340 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt"] Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978226 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978287 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978311 5233 cpu_manager.go:414] "containerMap: removing container" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerName="payload-pre-processing" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978319 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerName="payload-pre-processing" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978338 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978346 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978360 5233 cpu_manager.go:414] "containerMap: removing container" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978367 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" Nov 03 19:53:01.978369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978379 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978388 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978399 5233 cpu_manager.go:414] "containerMap: removing container" podUID="610b32be-07bd-4b82-89cb-ad4b63638981" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978406 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610b32be-07bd-4b82-89cb-ad4b63638981" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978427 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e6423373-89fc-451b-8b22-26e614285ad7" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978435 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6423373-89fc-451b-8b22-26e614285ad7" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978445 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a8705ab5-c573-4712-97f9-4bc7ec7d97fc" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978452 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8705ab5-c573-4712-97f9-4bc7ec7d97fc" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978478 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978484 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978506 5233 cpu_manager.go:414] "containerMap: removing container" podUID="8c8aece6-6558-4335-9410-18cfe23ec8b7" containerName="revoke-keys" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978513 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8aece6-6558-4335-9410-18cfe23ec8b7" containerName="revoke-keys" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978529 5233 cpu_manager.go:414] "containerMap: removing container" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978536 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978728 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98abe45-77ff-413a-8ea8-c88f9a7a26cd" containerName="main" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978744 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5540ea-5820-49ef-918f-655bcffeba0c" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978765 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f33cd4c-323e-4fe9-9820-a5e1b9342431" containerName="istio-proxy" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978779 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="86065665-60a4-4bce-89b6-3979cfe52fe9" containerName="main" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978793 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee48163e-ed96-4d75-8cc9-a86c1c0e35c5" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978820 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="610b32be-07bd-4b82-89cb-ad4b63638981" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978836 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6423373-89fc-451b-8b22-26e614285ad7" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978845 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8705ab5-c573-4712-97f9-4bc7ec7d97fc" containerName="payload-pre-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978854 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" containerName="payload-processing" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978865 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8aece6-6558-4335-9410-18cfe23ec8b7" containerName="revoke-keys" Nov 03 19:53:01.978960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.978875 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="319cda80-14b5-4529-8cc6-298f6b2e0353" containerName="main" Nov 03 19:53:01.983197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.983166 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt"] Nov 03 19:53:01.983391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.983310 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:01.997355 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.997204 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:01.998328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:01.997709 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:53:02.022862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.022756 5233 generic.go:356] "Generic (PLEG): container finished" podID="9359b24e-557f-428e-8205-5c9035c3f76e" containerID="a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5" exitCode=0 Nov 03 19:53:02.022862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.022813 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" Nov 03 19:53:02.022862 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.022838 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" event={"ID":"9359b24e-557f-428e-8205-5c9035c3f76e","Type":"ContainerDied","Data":"a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5"} Nov 03 19:53:02.023132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.022887 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69" event={"ID":"9359b24e-557f-428e-8205-5c9035c3f76e","Type":"ContainerDied","Data":"b4f7ba6b8b54ce80dc2cc026292f2afc99b1302593459333313717f251f54537"} Nov 03 19:53:02.023132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.022910 5233 scope.go:122] "RemoveContainer" containerID="a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5" Nov 03 19:53:02.030144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.028724 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.030144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.028787 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.030144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.028960 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28dg\" (UniqueName: \"kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.044527 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.044496 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69"] Nov 03 19:53:02.050010 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.049940 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-036184-676cfd8865-7fp69"] Nov 03 19:53:02.058454 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.058431 5233 scope.go:122] "RemoveContainer" containerID="a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5" Nov 03 19:53:02.058855 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:02.058831 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5\": container with ID starting with a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5 not found: ID does not exist" containerID="a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5" Nov 03 19:53:02.058920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.058864 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5"} err="failed to get container status \"a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5\": rpc error: code = NotFound desc = could not find container \"a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5\": container with ID starting with a220950bfb58d362f79eec9b3d5c77a274970c78964bfb0736431fdae537d5a5 not found: ID does not exist" Nov 03 19:53:02.130497 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.130459 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.130685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.130518 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.130685 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.130587 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28dg\" (UniqueName: \"kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.131198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.131174 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.132945 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.132922 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.143305 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.143278 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28dg\" (UniqueName: \"kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.304350 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.304243 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:02.601248 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:02.601218 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt"] Nov 03 19:53:02.602527 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:53:02.602500 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdf40f6_c83c_407c_9943_ae05587c80fb.slice/crio-710960e8d70c04e161bc4fa5f1183009f002ec904b3eeef3cdf254575e80dc93 WatchSource:0}: Error finding container 710960e8d70c04e161bc4fa5f1183009f002ec904b3eeef3cdf254575e80dc93: Status 404 returned error can't find the container with id 710960e8d70c04e161bc4fa5f1183009f002ec904b3eeef3cdf254575e80dc93 Nov 03 19:53:03.040955 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.040915 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" event={"ID":"6bdf40f6-c83c-407c-9943-ae05587c80fb","Type":"ContainerStarted","Data":"482ee59a2d5a7fa90ed941a4df1e805073759ad76aa4838f657c12ac521310fe"} Nov 03 19:53:03.041477 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.041178 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" event={"ID":"6bdf40f6-c83c-407c-9943-ae05587c80fb","Type":"ContainerStarted","Data":"710960e8d70c04e161bc4fa5f1183009f002ec904b3eeef3cdf254575e80dc93"} Nov 03 19:53:03.071714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.071658 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" podStartSLOduration=2.071643649 podStartE2EDuration="2.071643649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:03.067615058 +0000 UTC m=+1549.408728543" watchObservedRunningTime="2026-11-03 19:53:03.071643649 +0000 UTC m=+1549.412757149" Nov 03 19:53:03.360019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.357523 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w"] Nov 03 19:53:03.360019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.357773 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" containerID="cri-o://a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e" gracePeriod=30 Nov 03 19:53:03.363284 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.363253 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9"] Nov 03 19:53:03.363501 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.363476 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" containerID="cri-o://b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72" gracePeriod=30 Nov 03 19:53:03.837945 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.837911 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9359b24e-557f-428e-8205-5c9035c3f76e" path="/var/lib/kubelet/pods/9359b24e-557f-428e-8205-5c9035c3f76e/volumes" Nov 03 19:53:03.962216 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.962180 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2"] Nov 03 19:53:03.962459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.962409 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" containerID="cri-o://730976ec4e183b7307f63ea1fbd209f3a168399be2e550f2a556b26bf5c23247" gracePeriod=30 Nov 03 19:53:03.970169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.970134 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx"] Nov 03 19:53:03.970670 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.970579 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" containerID="cri-o://8fdc7329c54f76d09bf5f93e5dd82f38ac8321f8412677b4f6570b2c28f75fa4" gracePeriod=30 Nov 03 19:53:03.975414 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.975386 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq"] Nov 03 19:53:03.975607 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:03.975582 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" containerID="cri-o://c55c8ab0739495269844374fbf760d0e78a1d58b130a4e7cc323798a0164f8a3" gracePeriod=30 Nov 03 19:53:04.052857 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:04.052808 5233 generic.go:356] "Generic (PLEG): container finished" podID="6bdf40f6-c83c-407c-9943-ae05587c80fb" containerID="482ee59a2d5a7fa90ed941a4df1e805073759ad76aa4838f657c12ac521310fe" exitCode=0 Nov 03 19:53:04.053317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:04.052906 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" event={"ID":"6bdf40f6-c83c-407c-9943-ae05587c80fb","Type":"ContainerDied","Data":"482ee59a2d5a7fa90ed941a4df1e805073759ad76aa4838f657c12ac521310fe"} Nov 03 19:53:05.249935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.249908 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:05.294433 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.294390 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg\" (UniqueName: \"kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg\") pod \"6bdf40f6-c83c-407c-9943-ae05587c80fb\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " Nov 03 19:53:05.294631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.294515 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca\") pod \"6bdf40f6-c83c-407c-9943-ae05587c80fb\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " Nov 03 19:53:05.295366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.294671 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca\") pod \"6bdf40f6-c83c-407c-9943-ae05587c80fb\" (UID: \"6bdf40f6-c83c-407c-9943-ae05587c80fb\") " Nov 03 19:53:05.295366 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.295343 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca" pod "6bdf40f6-c83c-407c-9943-ae05587c80fb" (UID: "6bdf40f6-c83c-407c-9943-ae05587c80fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:05.297292 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.297263 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca" pod "6bdf40f6-c83c-407c-9943-ae05587c80fb" (UID: "6bdf40f6-c83c-407c-9943-ae05587c80fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:05.297633 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.297612 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg" pod "6bdf40f6-c83c-407c-9943-ae05587c80fb" (UID: "6bdf40f6-c83c-407c-9943-ae05587c80fb"). InnerVolumeSpecName "kube-api-access-m28dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:05.396351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.396243 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m28dg\" (UniqueName: \"kubernetes.io/projected/6bdf40f6-c83c-407c-9943-ae05587c80fb-kube-api-access-m28dg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:05.396351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.396286 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:05.396351 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:05.396302 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdf40f6-c83c-407c-9943-ae05587c80fb-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:06.089052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:06.089012 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" event={"ID":"6bdf40f6-c83c-407c-9943-ae05587c80fb","Type":"ContainerDied","Data":"710960e8d70c04e161bc4fa5f1183009f002ec904b3eeef3cdf254575e80dc93"} Nov 03 19:53:06.089331 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:06.089312 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710960e8d70c04e161bc4fa5f1183009f002ec904b3eeef3cdf254575e80dc93" Nov 03 19:53:06.089499 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:06.089487 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt" Nov 03 19:53:06.826593 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:06.826544 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:53:06.827553 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:06.827276 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:53:07.533660 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.533626 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh"] Nov 03 19:53:07.533905 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.533827 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" containerID="cri-o://d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d" gracePeriod=30 Nov 03 19:53:07.549938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.549908 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2"] Nov 03 19:53:07.550147 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.550125 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" containerID="cri-o://08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3" gracePeriod=30 Nov 03 19:53:07.560904 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.560878 5233 patch_prober.go:29] interesting pod/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:53:07.561045 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.560935 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:53:07.802071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.801961 5233 patch_prober.go:29] interesting pod/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:53:07.802071 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:07.802036 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:53:08.696689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.696218 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:53:08.698876 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.698851 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.747921 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.747991 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748040 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748087 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748136 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748166 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748193 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748311 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748346 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748436 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748472 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748507 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748548 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748576 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748622 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748656 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748719 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs\") pod \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\" (UID: \"cb41c5be-5f14-441b-bdbc-2cd395167c5d\") " Nov 03 19:53:08.749590 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.748757 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token\") pod \"c67ddf53-7a1c-43ae-9561-a6956711c692\" (UID: \"c67ddf53-7a1c-43ae-9561-a6956711c692\") " Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.751330 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.751580 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.751861 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.752183 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.752440 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.752642 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.752651 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.752822 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.753026 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.753139 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.753429 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.753345 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.754125 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.753527 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.755476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.755448 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "kube-api-access-mv2jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:08.764333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.764301 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:08.764333 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.764304 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo" pod "c67ddf53-7a1c-43ae-9561-a6956711c692" (UID: "c67ddf53-7a1c-43ae-9561-a6956711c692"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:08.764534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.764335 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:08.764534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.764385 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:08.765624 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.765601 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk" pod "cb41c5be-5f14-441b-bdbc-2cd395167c5d" (UID: "cb41c5be-5f14-441b-bdbc-2cd395167c5d"). InnerVolumeSpecName "kube-api-access-nb6rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:08.850185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850151 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850180 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67ddf53-7a1c-43ae-9561-a6956711c692-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850185 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850191 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850201 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850210 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850218 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850227 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850235 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850242 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850251 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850259 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850267 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb41c5be-5f14-441b-bdbc-2cd395167c5d-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850274 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850283 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2jt\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-kube-api-access-mv2jt\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850293 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6rk\" (UniqueName: \"kubernetes.io/projected/cb41c5be-5f14-441b-bdbc-2cd395167c5d-kube-api-access-nb6rk\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850301 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850309 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb41c5be-5f14-441b-bdbc-2cd395167c5d-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.850417 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:08.850317 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67ddf53-7a1c-43ae-9561-a6956711c692-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.047931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.047545 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7"] Nov 03 19:53:09.052866 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.052834 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb4e655-d5sd7"] Nov 03 19:53:09.146127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.144279 5233 generic.go:356] "Generic (PLEG): container finished" podID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerID="c55c8ab0739495269844374fbf760d0e78a1d58b130a4e7cc323798a0164f8a3" exitCode=0 Nov 03 19:53:09.146127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.144359 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" event={"ID":"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5","Type":"ContainerDied","Data":"c55c8ab0739495269844374fbf760d0e78a1d58b130a4e7cc323798a0164f8a3"} Nov 03 19:53:09.153030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.152469 5233 generic.go:356] "Generic (PLEG): container finished" podID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerID="a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e" exitCode=0 Nov 03 19:53:09.160665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.160131 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" event={"ID":"cb41c5be-5f14-441b-bdbc-2cd395167c5d","Type":"ContainerDied","Data":"a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e"} Nov 03 19:53:09.160665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.160215 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" event={"ID":"cb41c5be-5f14-441b-bdbc-2cd395167c5d","Type":"ContainerDied","Data":"d60308cccb862d53112db81cbded01468abb18f0468c8eabe0e9c33b8de99641"} Nov 03 19:53:09.160665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.160248 5233 scope.go:122] "RemoveContainer" containerID="a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e" Nov 03 19:53:09.160665 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.160440 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w" Nov 03 19:53:09.172507 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.172431 5233 generic.go:356] "Generic (PLEG): container finished" podID="2baec845-1de4-466c-82e8-2920589ba88d" containerID="8fdc7329c54f76d09bf5f93e5dd82f38ac8321f8412677b4f6570b2c28f75fa4" exitCode=0 Nov 03 19:53:09.172672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.172569 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" event={"ID":"2baec845-1de4-466c-82e8-2920589ba88d","Type":"ContainerDied","Data":"8fdc7329c54f76d09bf5f93e5dd82f38ac8321f8412677b4f6570b2c28f75fa4"} Nov 03 19:53:09.177676 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.177200 5233 generic.go:356] "Generic (PLEG): container finished" podID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerID="730976ec4e183b7307f63ea1fbd209f3a168399be2e550f2a556b26bf5c23247" exitCode=0 Nov 03 19:53:09.178744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.178621 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" event={"ID":"353f8791-bbf9-4b2b-97c9-9e6695a03721","Type":"ContainerDied","Data":"730976ec4e183b7307f63ea1fbd209f3a168399be2e550f2a556b26bf5c23247"} Nov 03 19:53:09.185824 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.185805 5233 scope.go:122] "RemoveContainer" containerID="a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e" Nov 03 19:53:09.186127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.186101 5233 generic.go:356] "Generic (PLEG): container finished" podID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerID="b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72" exitCode=0 Nov 03 19:53:09.186204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.186143 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" event={"ID":"c67ddf53-7a1c-43ae-9561-a6956711c692","Type":"ContainerDied","Data":"b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72"} Nov 03 19:53:09.186204 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.186177 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" event={"ID":"c67ddf53-7a1c-43ae-9561-a6956711c692","Type":"ContainerDied","Data":"fe0a75d49964a92ef627a901ba58ff3cdf6a0bad8d1e1c1c121c3b4d84ef9e4e"} Nov 03 19:53:09.186311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.186236 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9" Nov 03 19:53:09.192612 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:09.192578 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e\": container with ID starting with a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e not found: ID does not exist" containerID="a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e" Nov 03 19:53:09.192726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.192621 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e"} err="failed to get container status \"a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e\": rpc error: code = NotFound desc = could not find container \"a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e\": container with ID starting with a1f835328d287f59717237a1b3ec4b00fd0bbb37975a07c8134a1cee68cfae0e not found: ID does not exist" Nov 03 19:53:09.192726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.192645 5233 scope.go:122] "RemoveContainer" containerID="b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72" Nov 03 19:53:09.229848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.229156 5233 scope.go:122] "RemoveContainer" containerID="b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72" Nov 03 19:53:09.230702 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:09.230671 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72\": container with ID starting with b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72 not found: ID does not exist" containerID="b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72" Nov 03 19:53:09.230837 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.230713 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72"} err="failed to get container status \"b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72\": rpc error: code = NotFound desc = could not find container \"b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72\": container with ID starting with b922b85ae481b519f8c67f973ca6306e9bc92a7f7120f1a31a69b7c16e365e72 not found: ID does not exist" Nov 03 19:53:09.234026 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.234004 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w"] Nov 03 19:53:09.238488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.238473 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-65r6w"] Nov 03 19:53:09.246122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.246103 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9"] Nov 03 19:53:09.260186 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.260162 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-05e217-openshift-default-59c69f4bc9-lwlv9"] Nov 03 19:53:09.303147 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.301960 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:53:09.366052 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366026 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.366188 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366068 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.366369 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366346 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.366450 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366413 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.366506 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366467 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:09.366566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366509 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.366566 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366546 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.367012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366813 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.367012 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.366883 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.367172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.367031 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.367172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.367060 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.367172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.367096 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.367172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.367165 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx\") pod \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\" (UID: \"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5\") " Nov 03 19:53:09.367394 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.367261 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.368030 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.368005 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.368116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.368034 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.368116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.368050 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.368116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.368066 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.368116 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.368080 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.368336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.368300 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:53:09.369792 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.369768 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.370405 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.370386 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "kube-api-access-8jqcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.371330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.371310 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:09.372541 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.372522 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy" pod "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" (UID: "3e68eb51-7d02-4b99-8ff0-49d6319aa7a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.374127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.374105 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:53:09.469314 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469284 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.469480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469324 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.469480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469353 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.469480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469388 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.469480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469430 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.469480 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469454 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469503 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469537 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469582 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469624 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469656 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469661 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469690 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.469767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469741 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469792 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469818 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469827 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469862 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469889 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469902 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket\") pod \"353f8791-bbf9-4b2b-97c9-9e6695a03721\" (UID: \"353f8791-bbf9-4b2b-97c9-9e6695a03721\") " Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.469920 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470117 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470129 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket\") pod \"2baec845-1de4-466c-82e8-2920589ba88d\" (UID: \"2baec845-1de4-466c-82e8-2920589ba88d\") " Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470211 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.470245 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470234 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470705 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470729 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470745 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470760 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jqcx\" (UniqueName: \"kubernetes.io/projected/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-kube-api-access-8jqcx\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470775 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baec845-1de4-466c-82e8-2920589ba88d-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470788 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470808 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470825 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.470842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470838 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353f8791-bbf9-4b2b-97c9-9e6695a03721-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.471335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470851 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.471335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470865 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.471335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470707 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.471335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.470883 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.471335 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.471098 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.472236 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.472202 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.472393 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.472358 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "kube-api-access-rd2t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.472502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.472405 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "kube-api-access-9mt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.472502 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.472433 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.472681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.472664 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.473120 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.473096 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo" pod "353f8791-bbf9-4b2b-97c9-9e6695a03721" (UID: "353f8791-bbf9-4b2b-97c9-9e6695a03721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:09.473658 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.473636 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:09.474152 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.474134 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy" pod "2baec845-1de4-466c-82e8-2920589ba88d" (UID: "2baec845-1de4-466c-82e8-2920589ba88d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.572068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.571966 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572025 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572036 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572046 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572055 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572067 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baec845-1de4-466c-82e8-2920589ba88d-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572080 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baec845-1de4-466c-82e8-2920589ba88d-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572090 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572098 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353f8791-bbf9-4b2b-97c9-9e6695a03721-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572106 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2t6\" (UniqueName: \"kubernetes.io/projected/2baec845-1de4-466c-82e8-2920589ba88d-kube-api-access-rd2t6\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.572364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.572117 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mt66\" (UniqueName: \"kubernetes.io/projected/353f8791-bbf9-4b2b-97c9-9e6695a03721-kube-api-access-9mt66\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.843090 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.843017 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" path="/var/lib/kubelet/pods/c67ddf53-7a1c-43ae-9561-a6956711c692/volumes" Nov 03 19:53:09.843964 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.843936 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" path="/var/lib/kubelet/pods/cb41c5be-5f14-441b-bdbc-2cd395167c5d/volumes" Nov 03 19:53:09.844565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:09.844545 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b" path="/var/lib/kubelet/pods/d6ae4820-54c7-4ec5-b9f9-fbc50ac9558b/volumes" Nov 03 19:53:10.196943 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.196869 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" Nov 03 19:53:10.197140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.196868 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2" event={"ID":"353f8791-bbf9-4b2b-97c9-9e6695a03721","Type":"ContainerDied","Data":"f6e193a6c2dd1231ac2255293099be6393a28af58007923e7730f125c978a9e2"} Nov 03 19:53:10.197140 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.197048 5233 scope.go:122] "RemoveContainer" containerID="730976ec4e183b7307f63ea1fbd209f3a168399be2e550f2a556b26bf5c23247" Nov 03 19:53:10.200161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.200127 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" Nov 03 19:53:10.200161 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.200123 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq" event={"ID":"3e68eb51-7d02-4b99-8ff0-49d6319aa7a5","Type":"ContainerDied","Data":"03cea38257be6d0655319c03008cb85cbbb39e426e401bedf80722d892d1b1f7"} Nov 03 19:53:10.203291 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.203270 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" event={"ID":"2baec845-1de4-466c-82e8-2920589ba88d","Type":"ContainerDied","Data":"feb5d771537cb4a4573d58972373821dd14f506b9f66b19c41220d16e87dca18"} Nov 03 19:53:10.203391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.203290 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx" Nov 03 19:53:10.216468 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.216439 5233 scope.go:122] "RemoveContainer" containerID="c55c8ab0739495269844374fbf760d0e78a1d58b130a4e7cc323798a0164f8a3" Nov 03 19:53:10.228847 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.228685 5233 scope.go:122] "RemoveContainer" containerID="8fdc7329c54f76d09bf5f93e5dd82f38ac8321f8412677b4f6570b2c28f75fa4" Nov 03 19:53:10.234018 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.233963 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq"] Nov 03 19:53:10.237542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.237480 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-wl9fq"] Nov 03 19:53:10.248345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.248325 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2"] Nov 03 19:53:10.255304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.255279 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-6clt2"] Nov 03 19:53:10.269473 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.269449 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx"] Nov 03 19:53:10.275397 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:10.275370 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-036184-openshift-default-7bb76776f9-bn5qx"] Nov 03 19:53:11.837300 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:11.837270 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2baec845-1de4-466c-82e8-2920589ba88d" path="/var/lib/kubelet/pods/2baec845-1de4-466c-82e8-2920589ba88d/volumes" Nov 03 19:53:11.837930 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:11.837873 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" path="/var/lib/kubelet/pods/353f8791-bbf9-4b2b-97c9-9e6695a03721/volumes" Nov 03 19:53:11.838461 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:11.838441 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" path="/var/lib/kubelet/pods/3e68eb51-7d02-4b99-8ff0-49d6319aa7a5/volumes" Nov 03 19:53:12.007378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.007349 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv"] Nov 03 19:53:12.845015 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.844909 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt"] Nov 03 19:53:12.849464 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.849434 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-64454a-ws7bt"] Nov 03 19:53:12.920126 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.920082 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:53:12.943022 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.942888 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943025 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943064 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943142 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943213 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:12.943401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943221 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943360 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943444 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943450 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:12.943572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943490 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.943730 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.943602 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert\") pod \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\" (UID: \"fb82e649-4d20-4b2f-8457-deec1f0d0d9b\") " Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.944098 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.944436 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.944458 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.944473 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.944470 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.947233 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:12.961317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.948221 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:12.970663 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.970559 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:12.972885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.972840 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:12.973259 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.973068 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq" pod "fb82e649-4d20-4b2f-8457-deec1f0d0d9b" (UID: "fb82e649-4d20-4b2f-8457-deec1f0d0d9b"). InnerVolumeSpecName "kube-api-access-gqndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:12.993565 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:12.993513 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:53:13.045128 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045084 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045147 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045179 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045243 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045268 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045318 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045332 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:13.045542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045374 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045430 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045508 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token\") pod \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\" (UID: \"2a56a2fb-fc39-4103-b90e-adc6c07f4907\") " Nov 03 19:53:13.045542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045516 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:13.045840 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045610 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:13.045985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045898 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:13.045985 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.045937 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:13.046221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046135 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046158 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046174 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046186 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046199 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqndq\" (UniqueName: \"kubernetes.io/projected/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-kube-api-access-gqndq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046221 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046220 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046239 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046252 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb82e649-4d20-4b2f-8457-deec1f0d0d9b-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046265 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046279 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.046559 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.046293 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.047711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.047681 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:13.047851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.047827 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:13.048290 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.048269 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "kube-api-access-6wv6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:13.048364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.048329 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy" pod "2a56a2fb-fc39-4103-b90e-adc6c07f4907" (UID: "2a56a2fb-fc39-4103-b90e-adc6c07f4907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:13.147013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.146906 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.147013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.146931 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.147013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.146941 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wv6x\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-kube-api-access-6wv6x\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.147013 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.146954 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a56a2fb-fc39-4103-b90e-adc6c07f4907-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:13.236043 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.236011 5233 generic.go:356] "Generic (PLEG): container finished" podID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerID="08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3" exitCode=0 Nov 03 19:53:13.236210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.236099 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" event={"ID":"fb82e649-4d20-4b2f-8457-deec1f0d0d9b","Type":"ContainerDied","Data":"08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3"} Nov 03 19:53:13.236210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.236105 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" Nov 03 19:53:13.236210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.236151 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2" event={"ID":"fb82e649-4d20-4b2f-8457-deec1f0d0d9b","Type":"ContainerDied","Data":"3fa76012f6757491937e77d1f86e21553675b143b220a75aad63f8bf1dc8212b"} Nov 03 19:53:13.236210 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.236168 5233 scope.go:122] "RemoveContainer" containerID="08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3" Nov 03 19:53:13.238241 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.238209 5233 generic.go:356] "Generic (PLEG): container finished" podID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerID="d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d" exitCode=0 Nov 03 19:53:13.238378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.238255 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" Nov 03 19:53:13.238378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.238269 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" event={"ID":"2a56a2fb-fc39-4103-b90e-adc6c07f4907","Type":"ContainerDied","Data":"d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d"} Nov 03 19:53:13.238378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.238301 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh" event={"ID":"2a56a2fb-fc39-4103-b90e-adc6c07f4907","Type":"ContainerDied","Data":"7f17eb9538ec06310bae3359b5ebbd7c3e542e3b47fd93693b7a641a5d26133d"} Nov 03 19:53:13.249914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.249896 5233 scope.go:122] "RemoveContainer" containerID="08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3" Nov 03 19:53:13.250204 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:13.250187 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3\": container with ID starting with 08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3 not found: ID does not exist" containerID="08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3" Nov 03 19:53:13.250261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.250213 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3"} err="failed to get container status \"08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3\": rpc error: code = NotFound desc = could not find container \"08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3\": container with ID starting with 08f69884e5c2cc75061f467ad139b7101cdeaac3a2aa4f730fdf415c5c69eba3 not found: ID does not exist" Nov 03 19:53:13.250261 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.250229 5233 scope.go:122] "RemoveContainer" containerID="d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d" Nov 03 19:53:13.260691 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.260677 5233 scope.go:122] "RemoveContainer" containerID="f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b" Nov 03 19:53:13.271903 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.271870 5233 scope.go:122] "RemoveContainer" containerID="d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d" Nov 03 19:53:13.274322 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.274302 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh"] Nov 03 19:53:13.274576 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:13.274554 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d\": container with ID starting with d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d not found: ID does not exist" containerID="d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d" Nov 03 19:53:13.274638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.274585 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d"} err="failed to get container status \"d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d\": rpc error: code = NotFound desc = could not find container \"d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d\": container with ID starting with d9b4cbc2edaa6121a271d5105967b29dd25317fe21caaee320ac46c9a3887d2d not found: ID does not exist" Nov 03 19:53:13.274638 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.274603 5233 scope.go:122] "RemoveContainer" containerID="f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b" Nov 03 19:53:13.274909 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:13.274887 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b\": container with ID starting with f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b not found: ID does not exist" containerID="f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b" Nov 03 19:53:13.274962 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.274915 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b"} err="failed to get container status \"f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b\": rpc error: code = NotFound desc = could not find container \"f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b\": container with ID starting with f3aee944cdb291744f27aba7e76cce500200eee2889f27169d64c045638a033b not found: ID does not exist" Nov 03 19:53:13.281406 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.281377 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-tnjxh"] Nov 03 19:53:13.295881 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.295837 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2"] Nov 03 19:53:13.301470 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.301440 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-435bd8-openshift-default-6698b8f7c-465k2"] Nov 03 19:53:13.842534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.842457 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" path="/var/lib/kubelet/pods/2a56a2fb-fc39-4103-b90e-adc6c07f4907/volumes" Nov 03 19:53:13.843240 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.843213 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdf40f6-c83c-407c-9943-ae05587c80fb" path="/var/lib/kubelet/pods/6bdf40f6-c83c-407c-9943-ae05587c80fb/volumes" Nov 03 19:53:13.845198 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:13.845172 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" path="/var/lib/kubelet/pods/fb82e649-4d20-4b2f-8457-deec1f0d0d9b/volumes" Nov 03 19:53:17.020159 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.019901 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" podUID="15e58196-141a-4d6f-9815-034319ba770b" containerName="payload-processing" containerID="cri-o://8db769c1d70764f201ef9015480704e743475ba3e71dfa0533cb7d4c9ff06864" gracePeriod=25 Nov 03 19:53:17.282848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.282742 5233 generic.go:356] "Generic (PLEG): container finished" podID="15e58196-141a-4d6f-9815-034319ba770b" containerID="8db769c1d70764f201ef9015480704e743475ba3e71dfa0533cb7d4c9ff06864" exitCode=0 Nov 03 19:53:17.282848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.282822 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" event={"ID":"15e58196-141a-4d6f-9815-034319ba770b","Type":"ContainerDied","Data":"8db769c1d70764f201ef9015480704e743475ba3e71dfa0533cb7d4c9ff06864"} Nov 03 19:53:17.366635 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.366607 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:53:17.424423 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.424382 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume\") pod \"15e58196-141a-4d6f-9815-034319ba770b\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " Nov 03 19:53:17.424651 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.424623 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd\" (UniqueName: \"kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd\") pod \"15e58196-141a-4d6f-9815-034319ba770b\" (UID: \"15e58196-141a-4d6f-9815-034319ba770b\") " Nov 03 19:53:17.424823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.424799 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume" pod "15e58196-141a-4d6f-9815-034319ba770b" (UID: "15e58196-141a-4d6f-9815-034319ba770b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:17.425046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.425027 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e58196-141a-4d6f-9815-034319ba770b-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:17.427311 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.427283 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd" pod "15e58196-141a-4d6f-9815-034319ba770b" (UID: "15e58196-141a-4d6f-9815-034319ba770b"). InnerVolumeSpecName "kube-api-access-jfjfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:17.526701 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:17.526665 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfjfd\" (UniqueName: \"kubernetes.io/projected/15e58196-141a-4d6f-9815-034319ba770b-kube-api-access-jfjfd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:18.293822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:18.293793 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" Nov 03 19:53:18.294330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:18.293834 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv" event={"ID":"15e58196-141a-4d6f-9815-034319ba770b","Type":"ContainerDied","Data":"ecd6437b203d000dbd2efa4b9b0aa288ba8f3b0053ff5d8705eda1a57c2c678d"} Nov 03 19:53:18.294330 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:18.293911 5233 scope.go:122] "RemoveContainer" containerID="8db769c1d70764f201ef9015480704e743475ba3e71dfa0533cb7d4c9ff06864" Nov 03 19:53:18.315966 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:18.315938 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv"] Nov 03 19:53:18.330595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:18.330566 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-64454a-68c676dd75-8tmdv"] Nov 03 19:53:19.838091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:19.838055 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e58196-141a-4d6f-9815-034319ba770b" path="/var/lib/kubelet/pods/15e58196-141a-4d6f-9815-034319ba770b/volumes" Nov 03 19:53:21.153646 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.153611 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx"] Nov 03 19:53:21.154180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.153893 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" containerID="cri-o://02aee77e5f3a5c9a1365dec5edfac02d56e64b0500f4b9033be5f6b2f8b856ff" gracePeriod=30 Nov 03 19:53:21.277916 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:21.277880 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:53:21.695574 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.695538 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7"] Nov 03 19:53:21.696038 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696025 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696039 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696049 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696054 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696065 5233 cpu_manager.go:414] "containerMap: removing container" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696070 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696083 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696088 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696100 5233 cpu_manager.go:414] "containerMap: removing container" podUID="15e58196-141a-4d6f-9815-034319ba770b" containerName="payload-processing" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696105 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e58196-141a-4d6f-9815-034319ba770b" containerName="payload-processing" Nov 03 19:53:21.696112 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696114 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6bdf40f6-c83c-407c-9943-ae05587c80fb" containerName="revoke-keys" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696120 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdf40f6-c83c-407c-9943-ae05587c80fb" containerName="revoke-keys" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696129 5233 cpu_manager.go:414] "containerMap: removing container" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696134 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696154 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696160 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696166 5233 cpu_manager.go:414] "containerMap: removing container" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696170 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696178 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696182 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696243 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdf40f6-c83c-407c-9943-ae05587c80fb" containerName="revoke-keys" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696251 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e68eb51-7d02-4b99-8ff0-49d6319aa7a5" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696261 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67ddf53-7a1c-43ae-9561-a6956711c692" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696266 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82e649-4d20-4b2f-8457-deec1f0d0d9b" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696273 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baec845-1de4-466c-82e8-2920589ba88d" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696279 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696284 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a56a2fb-fc39-4103-b90e-adc6c07f4907" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696290 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="353f8791-bbf9-4b2b-97c9-9e6695a03721" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696295 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb41c5be-5f14-441b-bdbc-2cd395167c5d" containerName="istio-proxy" Nov 03 19:53:21.696401 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.696301 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e58196-141a-4d6f-9815-034319ba770b" containerName="payload-processing" Nov 03 19:53:21.700644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.700622 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.708307 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.708279 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7"] Nov 03 19:53:21.762597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.762566 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:21.762738 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.762620 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:53:21.783400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.783370 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.783532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.783407 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.783532 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.783436 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwt5j\" (UniqueName: \"kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.884522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.884484 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.884682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.884532 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.884682 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.884570 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwt5j\" (UniqueName: \"kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.885316 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.885280 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.886990 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.886954 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:21.898531 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:21.898508 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwt5j\" (UniqueName: \"kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:22.018002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:22.017980 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:22.319815 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:22.319786 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7"] Nov 03 19:53:22.321270 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:53:22.321242 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb10f0d_28b2_4e59_bf7f_7b9a0a30f994.slice/crio-5c3b704c22cf0deefc78335d49988b5e342c07a2f1d88c2d6f9241ae3600c220 WatchSource:0}: Error finding container 5c3b704c22cf0deefc78335d49988b5e342c07a2f1d88c2d6f9241ae3600c220: Status 404 returned error can't find the container with id 5c3b704c22cf0deefc78335d49988b5e342c07a2f1d88c2d6f9241ae3600c220 Nov 03 19:53:22.338303 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:22.338269 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" event={"ID":"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994","Type":"ContainerStarted","Data":"5c3b704c22cf0deefc78335d49988b5e342c07a2f1d88c2d6f9241ae3600c220"} Nov 03 19:53:23.347631 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:23.347549 5233 generic.go:356] "Generic (PLEG): container finished" podID="2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" containerID="211ad17feb57aea0e223523d831d72a9f2cddbfffb099bdc3a1a4680c23e25e4" exitCode=0 Nov 03 19:53:23.348024 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:23.347643 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" event={"ID":"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994","Type":"ContainerDied","Data":"211ad17feb57aea0e223523d831d72a9f2cddbfffb099bdc3a1a4680c23e25e4"} Nov 03 19:53:24.504820 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.504791 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:24.618422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.618335 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca\") pod \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " Nov 03 19:53:24.618422 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.618389 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j\" (UniqueName: \"kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j\") pod \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " Nov 03 19:53:24.618653 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.618536 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca\") pod \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\" (UID: \"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994\") " Nov 03 19:53:24.618960 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.618931 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca" pod "2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" (UID: "2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:24.620874 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.620843 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j" pod "2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" (UID: "2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994"). InnerVolumeSpecName "kube-api-access-wwt5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:24.621021 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.620955 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca" pod "2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" (UID: "2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:24.720274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.720227 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:24.720274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.720266 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:24.720274 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:24.720275 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwt5j\" (UniqueName: \"kubernetes.io/projected/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994-kube-api-access-wwt5j\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:25.365525 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:25.365489 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" Nov 03 19:53:25.365696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:25.365532 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7" event={"ID":"2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994","Type":"ContainerDied","Data":"5c3b704c22cf0deefc78335d49988b5e342c07a2f1d88c2d6f9241ae3600c220"} Nov 03 19:53:25.365696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:25.365642 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3b704c22cf0deefc78335d49988b5e342c07a2f1d88c2d6f9241ae3600c220" Nov 03 19:53:26.375006 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.374945 5233 generic.go:356] "Generic (PLEG): container finished" podID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerID="02aee77e5f3a5c9a1365dec5edfac02d56e64b0500f4b9033be5f6b2f8b856ff" exitCode=0 Nov 03 19:53:26.375371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.375029 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" event={"ID":"20b36ad6-bc31-4798-a6b5-e8333c37deb0","Type":"ContainerDied","Data":"02aee77e5f3a5c9a1365dec5edfac02d56e64b0500f4b9033be5f6b2f8b856ff"} Nov 03 19:53:26.450064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.449996 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:53:26.545097 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545068 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545104 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545134 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545197 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545252 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545323 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545372 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545419 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545414 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.545528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545450 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.545528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545459 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs\") pod \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\" (UID: \"20b36ad6-bc31-4798-a6b5-e8333c37deb0\") " Nov 03 19:53:26.545807 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545543 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.545957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545928 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.546119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.545964 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.546119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.546021 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.546119 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.546036 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.546232 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.546111 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:26.547789 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.547764 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.547894 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.547785 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:26.548019 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.547963 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "kube-api-access-fhhzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:26.548166 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.548149 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token" pod "20b36ad6-bc31-4798-a6b5-e8333c37deb0" (UID: "20b36ad6-bc31-4798-a6b5-e8333c37deb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:26.647600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.647567 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.647600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.647599 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.647600 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.647609 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.647818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.647618 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b36ad6-bc31-4798-a6b5-e8333c37deb0-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.647818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.647628 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhhzv\" (UniqueName: \"kubernetes.io/projected/20b36ad6-bc31-4798-a6b5-e8333c37deb0-kube-api-access-fhhzv\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.647818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:26.647638 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b36ad6-bc31-4798-a6b5-e8333c37deb0-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:27.383630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:27.383600 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" Nov 03 19:53:27.384064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:27.383631 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx" event={"ID":"20b36ad6-bc31-4798-a6b5-e8333c37deb0","Type":"ContainerDied","Data":"a98f9152775eb53a6ad051014776e88fb6a897794f451c7663618c7fbbd399ec"} Nov 03 19:53:27.384064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:27.383690 5233 scope.go:122] "RemoveContainer" containerID="02aee77e5f3a5c9a1365dec5edfac02d56e64b0500f4b9033be5f6b2f8b856ff" Nov 03 19:53:27.448416 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:27.448379 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx"] Nov 03 19:53:27.451379 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:27.451357 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-64454a-openshift-default-84c845c549-zrqrx"] Nov 03 19:53:27.835707 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:27.835666 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" path="/var/lib/kubelet/pods/20b36ad6-bc31-4798-a6b5-e8333c37deb0/volumes" Nov 03 19:53:31.851713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:31.851683 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv"] Nov 03 19:53:32.968719 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.968689 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5"] Nov 03 19:53:32.969239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.969218 5233 cpu_manager.go:414] "containerMap: removing container" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" Nov 03 19:53:32.969239 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.969240 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" Nov 03 19:53:32.969372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.969297 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" containerName="revoke-keys" Nov 03 19:53:32.969372 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.969304 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" containerName="revoke-keys" Nov 03 19:53:32.969438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.969391 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" containerName="revoke-keys" Nov 03 19:53:32.969438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.969403 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b36ad6-bc31-4798-a6b5-e8333c37deb0" containerName="istio-proxy" Nov 03 19:53:32.972871 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.972853 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:32.978184 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.978160 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:32.979543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.979517 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:53:32.983415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:32.983386 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5"] Nov 03 19:53:33.033046 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.033009 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72sk9\" (UniqueName: \"kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.033214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.033059 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.033214 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.033157 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.134059 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.134025 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72sk9\" (UniqueName: \"kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.134267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.134071 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.134267 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.134216 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.134791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.134765 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.136520 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.136498 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.146614 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.146590 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72sk9\" (UniqueName: \"kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9\") pod \"maas-api-revoke-sub-777363dc-4sxk5\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.289196 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.289112 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:33.575144 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:33.575116 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5"] Nov 03 19:53:33.576780 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:53:33.576746 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001594da_6aa5_4cf7_9514_a3e2d9ac7cdf.slice/crio-e8eb838aeb16ae281ef63d797a539a5cdb5afa80c89b5c296a3f5f3802738819 WatchSource:0}: Error finding container e8eb838aeb16ae281ef63d797a539a5cdb5afa80c89b5c296a3f5f3802738819: Status 404 returned error can't find the container with id e8eb838aeb16ae281ef63d797a539a5cdb5afa80c89b5c296a3f5f3802738819 Nov 03 19:53:34.438746 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:34.438712 5233 generic.go:356] "Generic (PLEG): container finished" podID="001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" containerID="6d997dcf8d81a004ec20d726151e38e0346d3d346735d43e96396db392a2936a" exitCode=0 Nov 03 19:53:34.439178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:34.438812 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" event={"ID":"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf","Type":"ContainerDied","Data":"6d997dcf8d81a004ec20d726151e38e0346d3d346735d43e96396db392a2936a"} Nov 03 19:53:34.439178 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:34.438880 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" event={"ID":"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf","Type":"ContainerStarted","Data":"e8eb838aeb16ae281ef63d797a539a5cdb5afa80c89b5c296a3f5f3802738819"} Nov 03 19:53:35.616830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.616803 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:35.662518 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.662475 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca\") pod \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " Nov 03 19:53:35.662714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.662533 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9\" (UniqueName: \"kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9\") pod \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " Nov 03 19:53:35.662714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.662570 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca\") pod \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\" (UID: \"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf\") " Nov 03 19:53:35.663465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.663436 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca" pod "001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" (UID: "001594da-6aa5-4cf7-9514-a3e2d9ac7cdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:35.666164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.666134 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9" pod "001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" (UID: "001594da-6aa5-4cf7-9514-a3e2d9ac7cdf"). InnerVolumeSpecName "kube-api-access-72sk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:35.667050 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.667028 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca" pod "001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" (UID: "001594da-6aa5-4cf7-9514-a3e2d9ac7cdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:35.763918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.763882 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:35.763918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.763913 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:35.763918 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:35.763923 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72sk9\" (UniqueName: \"kubernetes.io/projected/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf-kube-api-access-72sk9\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:36.458747 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:36.458714 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" Nov 03 19:53:36.458935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:36.458742 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5" event={"ID":"001594da-6aa5-4cf7-9514-a3e2d9ac7cdf","Type":"ContainerDied","Data":"e8eb838aeb16ae281ef63d797a539a5cdb5afa80c89b5c296a3f5f3802738819"} Nov 03 19:53:36.458935 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:36.458793 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8eb838aeb16ae281ef63d797a539a5cdb5afa80c89b5c296a3f5f3802738819" Nov 03 19:53:36.852262 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:36.852224 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" containerName="payload-pre-processing" containerID="cri-o://8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6" gracePeriod=25 Nov 03 19:53:45.097643 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:45.097600 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:53:48.197611 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:48.197576 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7"] Nov 03 19:53:48.201724 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:48.201697 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a4ea-mckl7"] Nov 03 19:53:49.863485 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:49.863450 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994" path="/var/lib/kubelet/pods/2bb10f0d-28b2-4e59-bf7f-7b9a0a30f994/volumes" Nov 03 19:53:51.277547 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:51.277506 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:53:55.098469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.098430 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:53:55.216259 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.216206 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p"] Nov 03 19:53:55.216517 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.216477 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" containerID="cri-o://f84ee0b87f868f3e873d24b5ef07da0f374e354210716f00c2481067215a5704" gracePeriod=30 Nov 03 19:53:55.219364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.219335 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4"] Nov 03 19:53:55.219597 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.219574 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" containerID="cri-o://3f4cdeaf4e2ad298769772f47d15d520abef9fbf58bf3ecd6cf57603a7eff2c0" gracePeriod=30 Nov 03 19:53:55.222150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.222125 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66"] Nov 03 19:53:55.222345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:55.222319 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" containerID="cri-o://dcb8990081a840a9914ad6c2b6adf73caf91a4eb3b477b3efdec8f214090d069" gracePeriod=30 Nov 03 19:53:57.125721 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.125688 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:53:57.264595 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.264563 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume\") pod \"c6204480-8a38-4886-8b54-fc88eb69902e\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " Nov 03 19:53:57.264764 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.264612 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf\" (UniqueName: \"kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf\") pod \"c6204480-8a38-4886-8b54-fc88eb69902e\" (UID: \"c6204480-8a38-4886-8b54-fc88eb69902e\") " Nov 03 19:53:57.264917 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.264895 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume" pod "c6204480-8a38-4886-8b54-fc88eb69902e" (UID: "c6204480-8a38-4886-8b54-fc88eb69902e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:57.265158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.265136 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6204480-8a38-4886-8b54-fc88eb69902e-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:57.266756 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.266736 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf" pod "c6204480-8a38-4886-8b54-fc88eb69902e" (UID: "c6204480-8a38-4886-8b54-fc88eb69902e"). InnerVolumeSpecName "kube-api-access-s8dwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:57.366657 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.366586 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8dwf\" (UniqueName: \"kubernetes.io/projected/c6204480-8a38-4886-8b54-fc88eb69902e-kube-api-access-s8dwf\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:57.650158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.650081 5233 generic.go:356] "Generic (PLEG): container finished" podID="c6204480-8a38-4886-8b54-fc88eb69902e" containerID="8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6" exitCode=0 Nov 03 19:53:57.650158 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.650128 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" Nov 03 19:53:57.650344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.650168 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" event={"ID":"c6204480-8a38-4886-8b54-fc88eb69902e","Type":"ContainerDied","Data":"8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6"} Nov 03 19:53:57.650344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.650232 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv" event={"ID":"c6204480-8a38-4886-8b54-fc88eb69902e","Type":"ContainerDied","Data":"a3d2b9a2306e0aa920e4768953d555b9b76d77d65db1479f47b45f92e6173588"} Nov 03 19:53:57.650344 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.650262 5233 scope.go:122] "RemoveContainer" containerID="8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6" Nov 03 19:53:57.665512 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.665484 5233 scope.go:122] "RemoveContainer" containerID="8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6" Nov 03 19:53:57.666213 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:53:57.666189 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6\": container with ID starting with 8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6 not found: ID does not exist" containerID="8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6" Nov 03 19:53:57.666323 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.666223 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6"} err="failed to get container status \"8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6\": rpc error: code = NotFound desc = could not find container \"8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6\": container with ID starting with 8d9be383e7e8b08cfa98cab600657058c4a33154347cae07f75ec4928cf60db6 not found: ID does not exist" Nov 03 19:53:57.704873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.704839 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv"] Nov 03 19:53:57.726931 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.726904 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d8a4ea-84b46d6f47-gpwvv"] Nov 03 19:53:57.836171 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:57.836141 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" path="/var/lib/kubelet/pods/c6204480-8a38-4886-8b54-fc88eb69902e/volumes" Nov 03 19:53:58.315465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:58.315427 5233 patch_prober.go:29] interesting pod/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:53:58.315869 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:53:58.315484 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:54:00.952589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:00.952339 5233 generic.go:356] "Generic (PLEG): container finished" podID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerID="f84ee0b87f868f3e873d24b5ef07da0f374e354210716f00c2481067215a5704" exitCode=0 Nov 03 19:54:00.952589 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:00.952474 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" event={"ID":"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1","Type":"ContainerDied","Data":"f84ee0b87f868f3e873d24b5ef07da0f374e354210716f00c2481067215a5704"} Nov 03 19:54:01.002074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.001022 5233 generic.go:356] "Generic (PLEG): container finished" podID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerID="3f4cdeaf4e2ad298769772f47d15d520abef9fbf58bf3ecd6cf57603a7eff2c0" exitCode=0 Nov 03 19:54:01.002074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.001211 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" event={"ID":"602cc334-6a9b-41c0-8ac3-7779884a6de2","Type":"ContainerDied","Data":"3f4cdeaf4e2ad298769772f47d15d520abef9fbf58bf3ecd6cf57603a7eff2c0"} Nov 03 19:54:01.002074 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.001270 5233 scope.go:122] "RemoveContainer" containerID="aa25e985b942629694fec8becdeca0521b8500e64c44d02dedb36b35add4f8ab" Nov 03 19:54:01.073965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.062462 5233 generic.go:356] "Generic (PLEG): container finished" podID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerID="dcb8990081a840a9914ad6c2b6adf73caf91a4eb3b477b3efdec8f214090d069" exitCode=0 Nov 03 19:54:01.073965 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.062563 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" event={"ID":"2bac15b2-cd9a-4400-9c88-3efa632d65a4","Type":"ContainerDied","Data":"dcb8990081a840a9914ad6c2b6adf73caf91a4eb3b477b3efdec8f214090d069"} Nov 03 19:54:01.308842 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.308811 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577529 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577631 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577695 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577731 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577782 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577835 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577923 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.577950 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.596040 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.578003 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo\") pod \"602cc334-6a9b-41c0-8ac3-7779884a6de2\" (UID: \"602cc334-6a9b-41c0-8ac3-7779884a6de2\") " Nov 03 19:54:01.637515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.609054 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:01.637515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.625046 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:01.637515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.636176 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:01.637515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.637248 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:01.637515 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.637463 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:01.698748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.698605 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:01.698748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.698658 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:01.698748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.698677 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:01.698748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.698694 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:01.698748 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.698709 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602cc334-6a9b-41c0-8ac3-7779884a6de2-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:01.850913 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.850820 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:54:01.884234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.884124 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "kube-api-access-slzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:01.884234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.884207 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:01.884463 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.884441 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:01.884543 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.884498 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo" pod "602cc334-6a9b-41c0-8ac3-7779884a6de2" (UID: "602cc334-6a9b-41c0-8ac3-7779884a6de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:54:01.915994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.902352 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:54:01.915994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.888168 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:01.915994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:01.912054 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slzcd\" (UniqueName: \"kubernetes.io/projected/602cc334-6a9b-41c0-8ac3-7779884a6de2-kube-api-access-slzcd\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.013288 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.013441 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.013512 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.013562 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.013626 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.014035 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.039994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.014058 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602cc334-6a9b-41c0-8ac3-7779884a6de2-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.041172 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.040919 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.056991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.041333 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.108067 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.096534 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120328 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120390 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120429 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120460 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120509 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120544 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120574 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120642 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120680 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120788 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert\") pod \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\" (UID: \"2bac15b2-cd9a-4400-9c88-3efa632d65a4\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120827 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.121017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.120937 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.121034 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs\") pod \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\" (UID: \"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1\") " Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.121475 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.121494 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.121510 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.157496 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.158195 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.178321 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.180161 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.180214 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.194050 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.188209 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.194369 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267775 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267810 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267825 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267841 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267856 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267870 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.273005 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.267885 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.282833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.280993 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" event={"ID":"2bac15b2-cd9a-4400-9c88-3efa632d65a4","Type":"ContainerDied","Data":"444f493fdf13e7332c8a6d831ffcd7964e57ac9edd4665ebaa24d51b90d1b37b"} Nov 03 19:54:02.282833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.281067 5233 scope.go:122] "RemoveContainer" containerID="dcb8990081a840a9914ad6c2b6adf73caf91a4eb3b477b3efdec8f214090d069" Nov 03 19:54:02.282833 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.281208 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66" Nov 03 19:54:02.357345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.285304 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" event={"ID":"cc83e3e7-a366-46d2-b351-7b8f23ffc5a1","Type":"ContainerDied","Data":"41819f01931a9f03c60fd7f72f7e88963ccb79be441c295f77fb8cfd4de0a511"} Nov 03 19:54:02.357345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.285393 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p" Nov 03 19:54:02.357345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.316367 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" event={"ID":"602cc334-6a9b-41c0-8ac3-7779884a6de2","Type":"ContainerDied","Data":"df58f374d87340bfbfa05fb700742121f09ed93096a53d40b099a1e45dea0e6d"} Nov 03 19:54:02.357345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.316465 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4" Nov 03 19:54:02.357345 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.334835 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "kube-api-access-zrvfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.367409 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.375644 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.375758 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.375826 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.377830 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.377855 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvfl\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-kube-api-access-zrvfl\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.377872 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.377887 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.378313 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.377901 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.422843 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.422253 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.423118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.423069 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy" pod "2bac15b2-cd9a-4400-9c88-3efa632d65a4" (UID: "2bac15b2-cd9a-4400-9c88-3efa632d65a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:02.450455 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.426817 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz" pod "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" (UID: "cc83e3e7-a366-46d2-b351-7b8f23ffc5a1"). InnerVolumeSpecName "kube-api-access-795dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:02.471830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.471740 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4"] Nov 03 19:54:02.471830 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.471798 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-658w4"] Nov 03 19:54:02.494302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.481311 5233 scope.go:122] "RemoveContainer" containerID="f84ee0b87f868f3e873d24b5ef07da0f374e354210716f00c2481067215a5704" Nov 03 19:54:02.494302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.486505 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.494302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.486533 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bac15b2-cd9a-4400-9c88-3efa632d65a4-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.494302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.486550 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795dz\" (UniqueName: \"kubernetes.io/projected/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1-kube-api-access-795dz\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:02.664785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.663156 5233 scope.go:122] "RemoveContainer" containerID="3f4cdeaf4e2ad298769772f47d15d520abef9fbf58bf3ecd6cf57603a7eff2c0" Nov 03 19:54:02.717008 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.709081 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p"] Nov 03 19:54:02.737465 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.725175 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-kng7p"] Nov 03 19:54:02.746742 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.746694 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66"] Nov 03 19:54:02.752058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:02.752005 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a4ea-openshift-default-6b47698d6c-dzx66"] Nov 03 19:54:03.836476 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:03.836444 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" path="/var/lib/kubelet/pods/2bac15b2-cd9a-4400-9c88-3efa632d65a4/volumes" Nov 03 19:54:03.836942 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:03.836926 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" path="/var/lib/kubelet/pods/602cc334-6a9b-41c0-8ac3-7779884a6de2/volumes" Nov 03 19:54:03.837430 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:03.837415 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" path="/var/lib/kubelet/pods/cc83e3e7-a366-46d2-b351-7b8f23ffc5a1/volumes" Nov 03 19:54:21.304474 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:54:21.304434 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:54:22.099914 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:22.099883 5233 scope.go:122] "RemoveContainer" containerID="84455cd07f63c970d9a1521744eced6ffc6c1ba52a8500821691cc7a668c0811" Nov 03 19:54:22.110498 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:22.110473 5233 scope.go:122] "RemoveContainer" containerID="20c3c0599c725723d3126b185fbe6cc51939d8f3f57c35e55ff216b97bb8228d" Nov 03 19:54:22.122744 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:22.122724 5233 scope.go:122] "RemoveContainer" containerID="4c171cd1f51dbd7f9abf636be8ba48c39cf7678992d4c8c6d62d6ae89985c5ac" Nov 03 19:54:22.132086 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:54:22.132065 5233 scope.go:122] "RemoveContainer" containerID="7665e3def4ebb3bc5d509d5de30ba2e883da09a3d7ad417c2ff9b54ec5c34b1d" Nov 03 19:54:51.276838 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:54:51.276799 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:55:21.275670 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:55:21.275636 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:55:22.226728 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:22.226690 5233 scope.go:122] "RemoveContainer" containerID="3f2afe26eb9189ee2f3662a502d3e5b756f6ee0ad2c5c415af6b2d28446ea11c" Nov 03 19:55:22.239672 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:22.239195 5233 scope.go:122] "RemoveContainer" containerID="9219317d05eb4b45affdc286c9af98ca1c5aad9e2ccf5abd1767ee6ffb250a1f" Nov 03 19:55:25.035968 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:25.035929 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm"] Nov 03 19:55:25.039276 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:25.039244 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087ec59d-t2bxm"] Nov 03 19:55:25.041987 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:25.041953 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5"] Nov 03 19:55:25.044215 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:25.044194 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce94f975-8xgz5"] Nov 03 19:55:25.836493 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:25.836461 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee3b83c-1446-4a17-b3ca-a4086b670c5a" path="/var/lib/kubelet/pods/6ee3b83c-1446-4a17-b3ca-a4086b670c5a/volumes" Nov 03 19:55:25.836886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:25.836873 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e86009-8112-45b6-a16a-854cedb6d0ee" path="/var/lib/kubelet/pods/d4e86009-8112-45b6-a16a-854cedb6d0ee/volumes" Nov 03 19:55:34.027127 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:34.027088 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs"] Nov 03 19:55:34.029572 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:34.029549 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f993953-zwjcs"] Nov 03 19:55:35.835644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:35.835611 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91640c01-d010-4792-9a3a-cbe633f8a85b" path="/var/lib/kubelet/pods/91640c01-d010-4792-9a3a-cbe633f8a85b/volumes" Nov 03 19:55:50.395474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:50.395080 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:55:50.395474 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:50.395163 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:50.576994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:50.576077 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:55:50.576994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:50.576158 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:50.619109 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:50.619068 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:55:50.619386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:50.619357 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.112003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:51.111074 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:55:51.112003 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:55:51.111154 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.305995 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:55:51.305516 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:56:07.940460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:07.940401 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4"] Nov 03 19:56:07.941055 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:07.940649 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" podUID="a0c59966-294c-4f89-9d00-d6077ec55c35" containerName="discovery" containerID="cri-o://b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55" gracePeriod=30 Nov 03 19:56:08.209767 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.209742 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:56:08.317192 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317160 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.317364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317228 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.317364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317333 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.317459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317371 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.317459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317393 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.317459 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317411 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.317641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.317495 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap\") pod \"a0c59966-294c-4f89-9d00-d6077ec55c35\" (UID: \"a0c59966-294c-4f89-9d00-d6077ec55c35\") " Nov 03 19:56:08.318302 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.318271 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:56:08.319677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.319652 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:08.319963 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.319938 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:56:08.320076 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.319958 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:56:08.320132 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.320079 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "kube-api-access-6qzdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:08.320169 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.320140 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:08.320592 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.320572 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts" pod "a0c59966-294c-4f89-9d00-d6077ec55c35" (UID: "a0c59966-294c-4f89-9d00-d6077ec55c35"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:56:08.419088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419052 5233 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-kubeconfig\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.419088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419083 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.419088 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419096 5233 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-dns-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.419338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419108 5233 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c59966-294c-4f89-9d00-d6077ec55c35-local-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.419338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419119 5233 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c59966-294c-4f89-9d00-d6077ec55c35-cacerts\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.419338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419131 5233 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c59966-294c-4f89-9d00-d6077ec55c35-istio-csr-ca-configmap\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.419338 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.419146 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzdw\" (UniqueName: \"kubernetes.io/projected/a0c59966-294c-4f89-9d00-d6077ec55c35-kube-api-access-6qzdw\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:08.554370 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.554343 5233 generic.go:356] "Generic (PLEG): container finished" podID="a0c59966-294c-4f89-9d00-d6077ec55c35" containerID="b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55" exitCode=0 Nov 03 19:56:08.554545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.554394 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" Nov 03 19:56:08.554545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.554431 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" event={"ID":"a0c59966-294c-4f89-9d00-d6077ec55c35","Type":"ContainerDied","Data":"b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55"} Nov 03 19:56:08.554545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.554481 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4" event={"ID":"a0c59966-294c-4f89-9d00-d6077ec55c35","Type":"ContainerDied","Data":"7478a98367cafd651a82dafc734542ece42afdac00b8bf96adb108cecb0ffdca"} Nov 03 19:56:08.554545 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.554501 5233 scope.go:122] "RemoveContainer" containerID="b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55" Nov 03 19:56:08.565785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.565768 5233 scope.go:122] "RemoveContainer" containerID="b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55" Nov 03 19:56:08.566086 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:56:08.566066 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55\": container with ID starting with b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55 not found: ID does not exist" containerID="b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55" Nov 03 19:56:08.566151 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.566093 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55"} err="failed to get container status \"b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55\": rpc error: code = NotFound desc = could not find container \"b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55\": container with ID starting with b148aa2ab9577c4ad29467e655d1dbca558c860ea9880bf36b7a2d02fb916e55 not found: ID does not exist" Nov 03 19:56:08.578961 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.578939 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4"] Nov 03 19:56:08.585817 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:08.585798 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8cp4"] Nov 03 19:56:09.835391 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:09.835364 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c59966-294c-4f89-9d00-d6077ec55c35" path="/var/lib/kubelet/pods/a0c59966-294c-4f89-9d00-d6077ec55c35/volumes" Nov 03 19:56:13.024639 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:13.024601 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm"] Nov 03 19:56:13.026999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:13.026963 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3957b994-8tlwm"] Nov 03 19:56:13.835561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:13.835531 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651471bd-0e0c-4a68-9ee3-8f0b9b678328" path="/var/lib/kubelet/pods/651471bd-0e0c-4a68-9ee3-8f0b9b678328/volumes" Nov 03 19:56:14.022207 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:14.022164 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb"] Nov 03 19:56:14.024785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:14.024758 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4175961-95dnb"] Nov 03 19:56:15.835848 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:15.835811 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7803a8fd-bd62-4358-a6e1-9961631b8ccd" path="/var/lib/kubelet/pods/7803a8fd-bd62-4358-a6e1-9961631b8ccd/volumes" Nov 03 19:56:21.276858 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:56:21.276822 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:56:22.267494 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:22.267460 5233 scope.go:122] "RemoveContainer" containerID="83e9228725fef8f6fbb63e1916cdad603c380fd63c9ba101e2b215353160291e" Nov 03 19:56:22.277242 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:22.277218 5233 scope.go:122] "RemoveContainer" containerID="f5541320c1f77fa767c09ce3fe5de965727e2df7ef0f5dfe1f6f06085b67054f" Nov 03 19:56:22.286661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:22.286645 5233 scope.go:122] "RemoveContainer" containerID="1d7a33fa137b0a35eb891d84edd0bd36e18ef337b50b44196798461bf487533e" Nov 03 19:56:22.295195 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:22.295167 5233 scope.go:122] "RemoveContainer" containerID="7abcbb52931e73a4d9699ee4553c5662e5589f19282ed41b483a3e81b2164cee" Nov 03 19:56:51.276321 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:56:51.276283 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:56:57.060713 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:57.060660 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx"] Nov 03 19:56:57.065677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:57.065650 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x"] Nov 03 19:56:57.070689 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:57.070647 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9031b44a-lspxx"] Nov 03 19:56:57.074364 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:57.074338 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5df7053b-sh86x"] Nov 03 19:56:57.853610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:57.853567 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c19abf-04cd-4b8c-8edc-88fa7c0bddda" path="/var/lib/kubelet/pods/32c19abf-04cd-4b8c-8edc-88fa7c0bddda/volumes" Nov 03 19:56:57.854252 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:56:57.854226 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f58bd3-f5bc-4b96-9203-9bfb637d58ee" path="/var/lib/kubelet/pods/68f58bd3-f5bc-4b96-9203-9bfb637d58ee/volumes" Nov 03 19:57:21.275777 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:57:21.275744 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:57:22.342482 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.342451 5233 scope.go:122] "RemoveContainer" containerID="193f29c6230b6a800400d978a171a8e0e49b0b1bdd40a29c2b3b61704ff15dbb" Nov 03 19:57:22.352064 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.352041 5233 scope.go:122] "RemoveContainer" containerID="660dbaa854240afc9865c54f07ba9b5ee5cc6ef434666ab389ea2666c870a45d" Nov 03 19:57:22.360763 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.360739 5233 scope.go:122] "RemoveContainer" containerID="d230b911e760cda5a57e3e5db2f84230b5ae0cc5fa0fa7b482b67fcb5772099f" Nov 03 19:57:22.369410 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.369393 5233 scope.go:122] "RemoveContainer" containerID="1cfab6d56d89477ceede86dcee79fa3e19c7979c87b6df85a0cd0b79397d53c2" Nov 03 19:57:22.377714 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.377699 5233 scope.go:122] "RemoveContainer" containerID="43f240845abca2ab64d19340c10196b94bc7b178e3f62c684739ed498b23a063" Nov 03 19:57:22.385823 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.385796 5233 scope.go:122] "RemoveContainer" containerID="e1552bbfec6dabcd70255bd22f15fbc03c49bf08ccec28031b8e15a8c1b128cb" Nov 03 19:57:22.397036 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.397020 5233 scope.go:122] "RemoveContainer" containerID="c346f57942a6b1df7ff22eecf717596177c8450a3ac6fdbc517d9788669ba836" Nov 03 19:57:22.405528 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:22.405505 5233 scope.go:122] "RemoveContainer" containerID="e2355e66d118814f96d00e5efe90843f13a2fb08ff4112a2f607c60d780c7fb4" Nov 03 19:57:35.575995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:35.575168 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:57:35.575995 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:35.575242 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:35.618994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:35.618067 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:57:35.618994 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:35.618130 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:36.158011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:36.157813 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:57:36.158011 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:36.157873 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:39.023827 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:39.023791 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx"] Nov 03 19:57:39.026272 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:39.026239 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e74dfcd-fqstx"] Nov 03 19:57:39.835745 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:39.835703 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6670ccde-86e8-4ceb-a292-40601cec7be5" path="/var/lib/kubelet/pods/6670ccde-86e8-4ceb-a292-40601cec7be5/volumes" Nov 03 19:57:44.024818 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:44.024781 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9"] Nov 03 19:57:44.027154 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:44.027092 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfbcaae4-tz4r9"] Nov 03 19:57:44.029523 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:44.029497 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9"] Nov 03 19:57:44.032164 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:44.032140 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584b57a5-sw6j9"] Nov 03 19:57:45.835403 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:45.835373 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1066d5cc-e41e-4e2d-9679-d83394ac4ffe" path="/var/lib/kubelet/pods/1066d5cc-e41e-4e2d-9679-d83394ac4ffe/volumes" Nov 03 19:57:45.835791 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:45.835762 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb449d2-bb35-4efd-8607-d703c21be1b4" path="/var/lib/kubelet/pods/beb449d2-bb35-4efd-8607-d703c21be1b4/volumes" Nov 03 19:57:49.024121 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:49.024082 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz"] Nov 03 19:57:49.028726 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:49.028700 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c393a05-2vxgz"] Nov 03 19:57:49.835797 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:57:49.835766 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8aece6-6558-4335-9410-18cfe23ec8b7" path="/var/lib/kubelet/pods/8c8aece6-6558-4335-9410-18cfe23ec8b7/volumes" Nov 03 19:57:51.280503 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:57:51.280454 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:58:05.575999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:05.575133 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:58:05.575999 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:05.575217 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:05.616856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:05.616418 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:58:05.616856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:05.616511 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:06.118000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:06.114091 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:58:06.118000 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:06.114204 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:21.277737 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:58:21.277702 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:58:22.456885 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.456851 5233 scope.go:122] "RemoveContainer" containerID="b4c27f8f96f718cffa0dc4b45a10c42e719687fa122be25616e2a4f7e55bf517" Nov 03 19:58:22.466400 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.466380 5233 scope.go:122] "RemoveContainer" containerID="e820024ba6811b5f432c43975570a1a47144fb52c27f8ba6cbf782197de769ca" Nov 03 19:58:22.475336 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.475318 5233 scope.go:122] "RemoveContainer" containerID="53f85b79ce0f70219230e22104f1f8906a45f36f78ff84d573409c56cd4de34a" Nov 03 19:58:22.486587 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.486560 5233 scope.go:122] "RemoveContainer" containerID="a30498d1d7c0d58f8db85d151153bf4c9c773cf1158ad7a821de5cce3ac9fba6" Nov 03 19:58:22.495993 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.495960 5233 scope.go:122] "RemoveContainer" containerID="8339db474e81510ca148cbfeab2d56d6aa4a5528bff647d4963c08d5eed23acf" Nov 03 19:58:22.504317 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.504299 5233 scope.go:122] "RemoveContainer" containerID="63d38b5cc1ed683e939614186d8b95148511c10e68705e3e8686e971bd2b5635" Nov 03 19:58:22.512122 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.512088 5233 scope.go:122] "RemoveContainer" containerID="a1568e54dd2d325bbb8fa066d12c89e9bfe5a7a1c67c9290e6a25cf2016a639c" Nov 03 19:58:22.519991 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.519955 5233 scope.go:122] "RemoveContainer" containerID="25b64575cd60217070363a79bb6aa566f364a8fb91cf79f4ceac03775ed549d8" Nov 03 19:58:22.528084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.527956 5233 scope.go:122] "RemoveContainer" containerID="ef6833943cd041da0e52e034d877fc35707577d7752e7f6ee8093bcb8ca861eb" Nov 03 19:58:22.535785 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:22.535764 5233 scope.go:122] "RemoveContainer" containerID="8229fbfb67fb4f7c6fd15082fd33722975ce8035cb3a038da44e96e14cd2d6e7" Nov 03 19:58:36.026873 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:36.026829 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5"] Nov 03 19:58:36.029911 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:36.029886 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-777363dc-4sxk5"] Nov 03 19:58:37.835630 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:37.835595 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" path="/var/lib/kubelet/pods/001594da-6aa5-4cf7-9514-a3e2d9ac7cdf/volumes" Nov 03 19:58:38.147469 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.147389 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz"] Nov 03 19:58:38.147637 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.147571 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" podUID="aef902ec-6022-4a16-be76-070d6d3f0f65" containerName="discovery" containerID="cri-o://1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30" gracePeriod=30 Nov 03 19:58:38.403925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.403868 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:58:38.467696 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.467669 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.467845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.467719 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.467845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.467764 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.467845 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.467831 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.467982 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.467922 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.467982 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.467956 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.468104 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.468011 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert\") pod \"aef902ec-6022-4a16-be76-070d6d3f0f65\" (UID: \"aef902ec-6022-4a16-be76-070d6d3f0f65\") " Nov 03 19:58:38.468197 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.468161 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:38.468467 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.468442 5233 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-ca-configmap\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:38.470057 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.470023 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:38.470194 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.470152 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:38.470386 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.470367 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:38.470444 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.470387 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "kube-api-access-8grst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:38.470483 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.470446 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:38.470725 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.470704 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert" pod "aef902ec-6022-4a16-be76-070d6d3f0f65" (UID: "aef902ec-6022-4a16-be76-070d6d3f0f65"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:38.570156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.570125 5233 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-cacerts\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:38.570156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.570150 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grst\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-kube-api-access-8grst\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:38.570156 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.570161 5233 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef902ec-6022-4a16-be76-070d6d3f0f65-local-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:38.570378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.570170 5233 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-kubeconfig\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:38.570378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.570180 5233 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-csr-dns-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:38.570378 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:38.570189 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef902ec-6022-4a16-be76-070d6d3f0f65-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:39.033264 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.033235 5233 generic.go:356] "Generic (PLEG): container finished" podID="aef902ec-6022-4a16-be76-070d6d3f0f65" containerID="1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30" exitCode=0 Nov 03 19:58:39.033693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.033297 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" Nov 03 19:58:39.033693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.033318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" event={"ID":"aef902ec-6022-4a16-be76-070d6d3f0f65","Type":"ContainerDied","Data":"1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30"} Nov 03 19:58:39.033693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.033368 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz" event={"ID":"aef902ec-6022-4a16-be76-070d6d3f0f65","Type":"ContainerDied","Data":"8c538bb30c005f9f91f86cfd550d8e77147ee02204b05d245f5690f1a935b5ab"} Nov 03 19:58:39.033693 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.033388 5233 scope.go:122] "RemoveContainer" containerID="1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30" Nov 03 19:58:39.044998 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.044960 5233 scope.go:122] "RemoveContainer" containerID="1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30" Nov 03 19:58:39.045288 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:58:39.045267 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30\": container with ID starting with 1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30 not found: ID does not exist" containerID="1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30" Nov 03 19:58:39.045349 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.045296 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30"} err="failed to get container status \"1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30\": rpc error: code = NotFound desc = could not find container \"1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30\": container with ID starting with 1702124d5cc4778b368770cc71125cd8b53bf5f4f9c68a0c7d54835837d4dc30 not found: ID does not exist" Nov 03 19:58:39.060893 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.060864 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz"] Nov 03 19:58:39.065542 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.065520 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tpjzz"] Nov 03 19:58:39.835234 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:39.835194 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef902ec-6022-4a16-be76-070d6d3f0f65" path="/var/lib/kubelet/pods/aef902ec-6022-4a16-be76-070d6d3f0f65/volumes" Nov 03 19:58:45.978957 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.978923 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx"] Nov 03 19:58:45.979582 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979561 5233 cpu_manager.go:414] "containerMap: removing container" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979584 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979596 5233 cpu_manager.go:414] "containerMap: removing container" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979603 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979630 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979639 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979658 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" containerName="payload-pre-processing" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979666 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" containerName="payload-pre-processing" Nov 03 19:58:45.979681 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979680 5233 cpu_manager.go:414] "containerMap: removing container" podUID="aef902ec-6022-4a16-be76-070d6d3f0f65" containerName="discovery" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979687 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef902ec-6022-4a16-be76-070d6d3f0f65" containerName="discovery" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979698 5233 cpu_manager.go:414] "containerMap: removing container" podUID="001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" containerName="revoke-keys" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979705 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" containerName="revoke-keys" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979727 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a0c59966-294c-4f89-9d00-d6077ec55c35" containerName="discovery" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979733 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c59966-294c-4f89-9d00-d6077ec55c35" containerName="discovery" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979846 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979859 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc83e3e7-a366-46d2-b351-7b8f23ffc5a1" containerName="istio-proxy" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979872 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bac15b2-cd9a-4400-9c88-3efa632d65a4" containerName="istio-proxy" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979882 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6204480-8a38-4886-8b54-fc88eb69902e" containerName="payload-pre-processing" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979891 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979900 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef902ec-6022-4a16-be76-070d6d3f0f65" containerName="discovery" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979909 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="001594da-6aa5-4cf7-9514-a3e2d9ac7cdf" containerName="revoke-keys" Nov 03 19:58:45.980118 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.979920 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c59966-294c-4f89-9d00-d6077ec55c35" containerName="discovery" Nov 03 19:58:45.984864 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.984837 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:45.990308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.990282 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:58:45.990736 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.990285 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:58:45.992522 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:45.992499 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx"] Nov 03 19:58:46.047720 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.047685 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.047920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.047748 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.047920 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.047832 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snh5q\" (UniqueName: \"kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.149206 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.149169 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.149415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.149221 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.149415 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.149248 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snh5q\" (UniqueName: \"kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.149986 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.149954 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.151621 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.151602 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.162058 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.162032 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snh5q\" (UniqueName: \"kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q\") pod \"maas-api-revoke-sub-5a29efb3-hqhpx\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.300798 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.300719 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:46.579694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:46.579665 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx"] Nov 03 19:58:46.581738 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:58:46.581708 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc687550b_4991_4ae6_9942_56425726b5fe.slice/crio-17b0c07b13a3a152e1fee49ce5264dc9d87644c02ae4ae7537a6061b4f9222a3 WatchSource:0}: Error finding container 17b0c07b13a3a152e1fee49ce5264dc9d87644c02ae4ae7537a6061b4f9222a3: Status 404 returned error can't find the container with id 17b0c07b13a3a152e1fee49ce5264dc9d87644c02ae4ae7537a6061b4f9222a3 Nov 03 19:58:47.084758 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:47.084715 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" event={"ID":"c687550b-4991-4ae6-9942-56425726b5fe","Type":"ContainerStarted","Data":"6282a598dd1e204d6c7146ae61841544c27e62fea87c8ae5b39914232ced34be"} Nov 03 19:58:47.085304 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:47.084783 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" event={"ID":"c687550b-4991-4ae6-9942-56425726b5fe","Type":"ContainerStarted","Data":"17b0c07b13a3a152e1fee49ce5264dc9d87644c02ae4ae7537a6061b4f9222a3"} Nov 03 19:58:47.108167 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:47.108095 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" podStartSLOduration=2.108074749 podStartE2EDuration="2.108074749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:47.104643073 +0000 UTC m=+1893.445756565" watchObservedRunningTime="2026-11-03 19:58:47.108074749 +0000 UTC m=+1893.449188235" Nov 03 19:58:48.147605 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:48.147567 5233 generic.go:356] "Generic (PLEG): container finished" podID="c687550b-4991-4ae6-9942-56425726b5fe" containerID="6282a598dd1e204d6c7146ae61841544c27e62fea87c8ae5b39914232ced34be" exitCode=0 Nov 03 19:58:48.148694 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:48.148479 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" event={"ID":"c687550b-4991-4ae6-9942-56425726b5fe","Type":"ContainerDied","Data":"6282a598dd1e204d6c7146ae61841544c27e62fea87c8ae5b39914232ced34be"} Nov 03 19:58:50.085365 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.085334 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:50.190180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.188790 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" event={"ID":"c687550b-4991-4ae6-9942-56425726b5fe","Type":"ContainerDied","Data":"17b0c07b13a3a152e1fee49ce5264dc9d87644c02ae4ae7537a6061b4f9222a3"} Nov 03 19:58:50.190180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.188841 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b0c07b13a3a152e1fee49ce5264dc9d87644c02ae4ae7537a6061b4f9222a3" Nov 03 19:58:50.190180 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.188898 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx" Nov 03 19:58:50.197488 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.197464 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca\") pod \"c687550b-4991-4ae6-9942-56425726b5fe\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " Nov 03 19:58:50.197647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.197536 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q\" (UniqueName: \"kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q\") pod \"c687550b-4991-4ae6-9942-56425726b5fe\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " Nov 03 19:58:50.197647 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.197593 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca\") pod \"c687550b-4991-4ae6-9942-56425726b5fe\" (UID: \"c687550b-4991-4ae6-9942-56425726b5fe\") " Nov 03 19:58:50.198007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.197962 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca" pod "c687550b-4991-4ae6-9942-56425726b5fe" (UID: "c687550b-4991-4ae6-9942-56425726b5fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:50.199938 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.199911 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca" pod "c687550b-4991-4ae6-9942-56425726b5fe" (UID: "c687550b-4991-4ae6-9942-56425726b5fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:50.200049 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.199941 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q" pod "c687550b-4991-4ae6-9942-56425726b5fe" (UID: "c687550b-4991-4ae6-9942-56425726b5fe"). InnerVolumeSpecName "kube-api-access-snh5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:50.299278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.299200 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.299278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.299231 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c687550b-4991-4ae6-9942-56425726b5fe-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.299278 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:58:50.299249 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snh5q\" (UniqueName: \"kubernetes.io/projected/c687550b-4991-4ae6-9942-56425726b5fe-kube-api-access-snh5q\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:51.276239 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:58:51.276206 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:59:02.077677 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.077635 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t"] Nov 03 19:59:02.078328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.078209 5233 cpu_manager.go:414] "containerMap: removing container" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" Nov 03 19:59:02.078328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.078231 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602cc334-6a9b-41c0-8ac3-7779884a6de2" containerName="istio-proxy" Nov 03 19:59:02.078328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.078270 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c687550b-4991-4ae6-9942-56425726b5fe" containerName="revoke-keys" Nov 03 19:59:02.078328 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.078275 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687550b-4991-4ae6-9942-56425726b5fe" containerName="revoke-keys" Nov 03 19:59:02.078503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.078356 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687550b-4991-4ae6-9942-56425726b5fe" containerName="revoke-keys" Nov 03 19:59:02.081886 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.081869 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.090002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.088769 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:59:02.090002 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.089916 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 19:59:02.091371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.091338 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t"] Nov 03 19:59:02.112534 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.112487 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.112709 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.112686 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.112775 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.112739 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvh8j\" (UniqueName: \"kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.213343 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.213313 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.213503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.213354 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvh8j\" (UniqueName: \"kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.213503 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.213398 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.213925 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.213904 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.215588 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.215569 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.225838 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.225815 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvh8j\" (UniqueName: \"kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j\") pod \"maas-api-revoke-sub-3d4156f1-vjc2t\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.403641 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.403560 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:02.682308 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:02.682161 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t"] Nov 03 19:59:02.685040 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:59:02.685008 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd882be8_2380_4be8_b135_13699c3331ff.slice/crio-f34f3123fb96e70bd09deedcb463bca7ba786d65dd85261e2d500192f5a6b840 WatchSource:0}: Error finding container f34f3123fb96e70bd09deedcb463bca7ba786d65dd85261e2d500192f5a6b840: Status 404 returned error can't find the container with id f34f3123fb96e70bd09deedcb463bca7ba786d65dd85261e2d500192f5a6b840 Nov 03 19:59:03.270560 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:03.270518 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" event={"ID":"dd882be8-2380-4be8-b135-13699c3331ff","Type":"ContainerStarted","Data":"69a718b46f41f0a3b6bb6a151eed3356a97bac460b90328fc5af07ce2f5dadbe"} Nov 03 19:59:03.271048 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:03.270577 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" event={"ID":"dd882be8-2380-4be8-b135-13699c3331ff","Type":"ContainerStarted","Data":"f34f3123fb96e70bd09deedcb463bca7ba786d65dd85261e2d500192f5a6b840"} Nov 03 19:59:03.290636 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:03.290588 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" podStartSLOduration=1.290575168 podStartE2EDuration="1.290575168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:03.289083465 +0000 UTC m=+1909.630196950" watchObservedRunningTime="2026-11-03 19:59:03.290575168 +0000 UTC m=+1909.631688653" Nov 03 19:59:04.277910 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:04.277874 5233 generic.go:356] "Generic (PLEG): container finished" podID="dd882be8-2380-4be8-b135-13699c3331ff" containerID="69a718b46f41f0a3b6bb6a151eed3356a97bac460b90328fc5af07ce2f5dadbe" exitCode=0 Nov 03 19:59:04.278334 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:04.277945 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" event={"ID":"dd882be8-2380-4be8-b135-13699c3331ff","Type":"ContainerDied","Data":"69a718b46f41f0a3b6bb6a151eed3356a97bac460b90328fc5af07ce2f5dadbe"} Nov 03 19:59:05.444711 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.444686 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:05.544888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.544811 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca\") pod \"dd882be8-2380-4be8-b135-13699c3331ff\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " Nov 03 19:59:05.544888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.544854 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j\" (UniqueName: \"kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j\") pod \"dd882be8-2380-4be8-b135-13699c3331ff\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " Nov 03 19:59:05.544888 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.544887 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca\") pod \"dd882be8-2380-4be8-b135-13699c3331ff\" (UID: \"dd882be8-2380-4be8-b135-13699c3331ff\") " Nov 03 19:59:05.545358 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.545331 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca" pod "dd882be8-2380-4be8-b135-13699c3331ff" (UID: "dd882be8-2380-4be8-b135-13699c3331ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:05.547084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.547058 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j" pod "dd882be8-2380-4be8-b135-13699c3331ff" (UID: "dd882be8-2380-4be8-b135-13699c3331ff"). InnerVolumeSpecName "kube-api-access-bvh8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:05.547084 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.547067 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca" pod "dd882be8-2380-4be8-b135-13699c3331ff" (UID: "dd882be8-2380-4be8-b135-13699c3331ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:59:05.646419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.646388 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:05.646419 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.646414 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvh8j\" (UniqueName: \"kubernetes.io/projected/dd882be8-2380-4be8-b135-13699c3331ff-kube-api-access-bvh8j\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:05.646594 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:05.646425 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd882be8-2380-4be8-b135-13699c3331ff-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:06.290644 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:06.290616 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" Nov 03 19:59:06.290822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:06.290656 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t" event={"ID":"dd882be8-2380-4be8-b135-13699c3331ff","Type":"ContainerDied","Data":"f34f3123fb96e70bd09deedcb463bca7ba786d65dd85261e2d500192f5a6b840"} Nov 03 19:59:06.290822 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:06.290709 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f34f3123fb96e70bd09deedcb463bca7ba786d65dd85261e2d500192f5a6b840" Nov 03 19:59:21.277544 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:59:21.277505 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 19:59:22.597460 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.597426 5233 scope.go:122] "RemoveContainer" containerID="211ad17feb57aea0e223523d831d72a9f2cddbfffb099bdc3a1a4680c23e25e4" Nov 03 19:59:22.607047 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.607025 5233 scope.go:122] "RemoveContainer" containerID="4f29edc7b2c84e317b2d8634dc4b7edacfd0bdf4afd6f48d345f3eaee4b80e72" Nov 03 19:59:22.615295 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.615278 5233 scope.go:122] "RemoveContainer" containerID="482ee59a2d5a7fa90ed941a4df1e805073759ad76aa4838f657c12ac521310fe" Nov 03 19:59:22.623091 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.623068 5233 scope.go:122] "RemoveContainer" containerID="1b05bc7e334f5c4eaa09b4e15ac4e4074e7a108f30d8cbbd8cf694dca78a1551" Nov 03 19:59:22.630794 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.630770 5233 scope.go:122] "RemoveContainer" containerID="ab19f8a3d109db66d234eaf68fbe046d0e7c703ba8c99d9cdfbe7b47aef803a1" Nov 03 19:59:22.638359 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.638341 5233 scope.go:122] "RemoveContainer" containerID="ef17ff0410871ca31d093216b5d1398238965c24ed6f2b52922482f9fd98b409" Nov 03 19:59:22.645851 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.645827 5233 scope.go:122] "RemoveContainer" containerID="3e2a82dbb08118792fe73ad0f3b1ef1610ce0c5d3b6801b18cdd626428ceb5f2" Nov 03 19:59:22.653243 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.653226 5233 scope.go:122] "RemoveContainer" containerID="49706dfc2f3d38a76458e9d2c60c4a453795b039460dfc5bfc3d5643314d096f" Nov 03 19:59:22.660645 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:22.660627 5233 scope.go:122] "RemoveContainer" containerID="382b63c26cee03b73e4a0b48862e1c599198ca12797edc284c0cb3a051302ab6" Nov 03 19:59:23.028828 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.028795 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh"] Nov 03 19:59:23.029277 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.029262 5233 cpu_manager.go:414] "containerMap: removing container" podUID="dd882be8-2380-4be8-b135-13699c3331ff" containerName="revoke-keys" Nov 03 19:59:23.029329 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.029279 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd882be8-2380-4be8-b135-13699c3331ff" containerName="revoke-keys" Nov 03 19:59:23.029371 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.029354 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd882be8-2380-4be8-b135-13699c3331ff" containerName="revoke-keys" Nov 03 19:59:23.034438 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.034416 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.047997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.047630 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh"] Nov 03 19:59:23.047997 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.047738 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-458393\"" Nov 03 19:59:23.048150 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.048003 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-71d2892b-dockercfg-m5jw6\"" Nov 03 19:59:23.089286 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.089261 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g"] Nov 03 19:59:23.097856 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.097831 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.117805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.117780 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g"] Nov 03 19:59:23.125573 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.125543 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4n72\" (UniqueName: \"kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72\") pod \"payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.125688 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.125595 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.226674 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.226641 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25clb\" (UniqueName: \"kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb\") pod \"payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.226854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.226792 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4n72\" (UniqueName: \"kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72\") pod \"payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.226854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.226826 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.226854 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.226852 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.227661 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.227635 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.239699 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.239670 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4n72\" (UniqueName: \"kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72\") pod \"payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.328183 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.328107 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25clb\" (UniqueName: \"kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb\") pod \"payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.328327 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.328229 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.328899 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.328879 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.341337 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.341311 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25clb\" (UniqueName: \"kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb\") pod \"payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.354160 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.354112 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:23.412882 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.411225 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:23.646497 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.646474 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh"] Nov 03 19:59:23.648713 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:59:23.648684 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e70130e_ba79_49e5_8d57_709fba1d75d7.slice/crio-4c0b43fbc0f45a337ce77ff6f27119a955459772a8f73e3feaf57e5372d2414f WatchSource:0}: Error finding container 4c0b43fbc0f45a337ce77ff6f27119a955459772a8f73e3feaf57e5372d2414f: Status 404 returned error can't find the container with id 4c0b43fbc0f45a337ce77ff6f27119a955459772a8f73e3feaf57e5372d2414f Nov 03 19:59:23.724301 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:23.724229 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g"] Nov 03 19:59:23.726420 ip-10-0-128-162 kubenswrapper[5233]: W1103 19:59:23.726394 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128dd3cc_0329_4d7f_b9c0_58a33a03dfad.slice/crio-f4e2b8d02303ed5ec713005e49276d8619ae83e7bd2544d9cf9dc640c6b08486 WatchSource:0}: Error finding container f4e2b8d02303ed5ec713005e49276d8619ae83e7bd2544d9cf9dc640c6b08486: Status 404 returned error can't find the container with id f4e2b8d02303ed5ec713005e49276d8619ae83e7bd2544d9cf9dc640c6b08486 Nov 03 19:59:24.404007 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.403957 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" event={"ID":"128dd3cc-0329-4d7f-b9c0-58a33a03dfad","Type":"ContainerStarted","Data":"df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4"} Nov 03 19:59:24.404238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.404029 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" event={"ID":"128dd3cc-0329-4d7f-b9c0-58a33a03dfad","Type":"ContainerStarted","Data":"f4e2b8d02303ed5ec713005e49276d8619ae83e7bd2544d9cf9dc640c6b08486"} Nov 03 19:59:24.404238 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.404162 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:24.405805 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.405778 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" event={"ID":"9e70130e-ba79-49e5-8d57-709fba1d75d7","Type":"ContainerStarted","Data":"d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837"} Nov 03 19:59:24.405951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.405827 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" event={"ID":"9e70130e-ba79-49e5-8d57-709fba1d75d7","Type":"ContainerStarted","Data":"4c0b43fbc0f45a337ce77ff6f27119a955459772a8f73e3feaf57e5372d2414f"} Nov 03 19:59:24.405951 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.405883 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:24.429610 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.429553 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" podStartSLOduration=1.429535684 podStartE2EDuration="1.429535684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:24.426228961 +0000 UTC m=+1930.767342445" watchObservedRunningTime="2026-11-03 19:59:24.429535684 +0000 UTC m=+1930.770649172" Nov 03 19:59:24.452017 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:24.451956 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" podStartSLOduration=1.451941093 podStartE2EDuration="1.451941093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:24.449982686 +0000 UTC m=+1930.791096172" watchObservedRunningTime="2026-11-03 19:59:24.451941093 +0000 UTC m=+1930.793054580" Nov 03 19:59:35.415561 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:35.415531 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 19:59:35.416068 ip-10-0-128-162 kubenswrapper[5233]: I1103 19:59:35.415600 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 19:59:51.276933 ip-10-0-128-162 kubenswrapper[5233]: E1103 19:59:51.276898 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:00:00.149436 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.149396 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g"] Nov 03 20:00:00.153789 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.153764 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.164620 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.164596 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:00:00.167303 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.167280 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g"] Nov 03 20:00:00.328133 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.328101 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xs2\" (UniqueName: \"kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2\") pod \"maas-api-key-cleanup-29895600-dtw7g\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.328343 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.328145 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-dtw7g\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.429735 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.429639 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xs2\" (UniqueName: \"kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2\") pod \"maas-api-key-cleanup-29895600-dtw7g\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.429735 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.429690 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-dtw7g\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.430448 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.430422 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-dtw7g\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.443826 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.443799 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xs2\" (UniqueName: \"kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2\") pod \"maas-api-key-cleanup-29895600-dtw7g\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.469585 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.469558 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:00.755713 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:00.755688 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g"] Nov 03 20:00:00.757744 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:00:00.757710 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dcb338_6ba6_4a7b_8b84_40befd7a216b.slice/crio-a375cf92e0e42910a77f14cc2c9e3cd0c57f902c8fad552d55016cc5a5316040 WatchSource:0}: Error finding container a375cf92e0e42910a77f14cc2c9e3cd0c57f902c8fad552d55016cc5a5316040: Status 404 returned error can't find the container with id a375cf92e0e42910a77f14cc2c9e3cd0c57f902c8fad552d55016cc5a5316040 Nov 03 20:00:01.634963 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:01.634938 5233 generic.go:356] "Generic (PLEG): container finished" podID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerID="eebd6057297a71d3b8b213466ff86312e4bfde198b51e68248bfe80fb20af0de" exitCode=60 Nov 03 20:00:01.635351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:01.635002 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" event={"ID":"b3dcb338-6ba6-4a7b-8b84-40befd7a216b","Type":"ContainerDied","Data":"eebd6057297a71d3b8b213466ff86312e4bfde198b51e68248bfe80fb20af0de"} Nov 03 20:00:01.635351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:01.635053 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" event={"ID":"b3dcb338-6ba6-4a7b-8b84-40befd7a216b","Type":"ContainerStarted","Data":"a375cf92e0e42910a77f14cc2c9e3cd0c57f902c8fad552d55016cc5a5316040"} Nov 03 20:00:01.635351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:01.635296 5233 scope.go:122] "RemoveContainer" containerID="eebd6057297a71d3b8b213466ff86312e4bfde198b51e68248bfe80fb20af0de" Nov 03 20:00:02.642893 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:02.642802 5233 generic.go:356] "Generic (PLEG): container finished" podID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerID="7305d03489095866b4503796150dd9fbb71f01ef2fc08cfb9b9a9426591486a3" exitCode=60 Nov 03 20:00:02.642893 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:02.642850 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" event={"ID":"b3dcb338-6ba6-4a7b-8b84-40befd7a216b","Type":"ContainerDied","Data":"7305d03489095866b4503796150dd9fbb71f01ef2fc08cfb9b9a9426591486a3"} Nov 03 20:00:02.643411 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:02.642910 5233 scope.go:122] "RemoveContainer" containerID="eebd6057297a71d3b8b213466ff86312e4bfde198b51e68248bfe80fb20af0de" Nov 03 20:00:02.643411 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:02.643205 5233 scope.go:122] "RemoveContainer" containerID="7305d03489095866b4503796150dd9fbb71f01ef2fc08cfb9b9a9426591486a3" Nov 03 20:00:02.643509 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:02.643466 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-dtw7g_odh-ai-gateway-infra(b3dcb338-6ba6-4a7b-8b84-40befd7a216b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" Nov 03 20:00:03.655410 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:03.655377 5233 scope.go:122] "RemoveContainer" containerID="7305d03489095866b4503796150dd9fbb71f01ef2fc08cfb9b9a9426591486a3" Nov 03 20:00:03.655795 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:03.655639 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-dtw7g_odh-ai-gateway-infra(b3dcb338-6ba6-4a7b-8b84-40befd7a216b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" Nov 03 20:00:08.720400 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.720367 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn"] Nov 03 20:00:08.724261 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.724243 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.729599 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.729570 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:08.733689 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.733669 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn"] Nov 03 20:00:08.810206 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.810169 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpph\" (UniqueName: \"kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.810375 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.810278 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.810375 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.810352 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.911203 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.911168 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.911385 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.911220 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpph\" (UniqueName: \"kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.911385 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.911272 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.911860 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.911839 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.913548 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.913523 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:08.925934 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:08.925908 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpph\" (UniqueName: \"kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph\") pod \"maas-api-revoke-sub-b306a730-jkjjn\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:09.039908 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:09.039841 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:09.316904 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:09.316863 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn"] Nov 03 20:00:09.692291 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:09.692251 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" event={"ID":"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa","Type":"ContainerStarted","Data":"4a32091e416beaada586356f3ae95aee90c6c058ce58e4e77254142f844204fa"} Nov 03 20:00:09.692440 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:09.692318 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" event={"ID":"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa","Type":"ContainerStarted","Data":"fd972923593ac8eb8ae4e99ebdf734d70b649627dff15880e8a70809b8f0edb1"} Nov 03 20:00:09.712794 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:09.712743 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" podStartSLOduration=1.712730086 podStartE2EDuration="1.712730086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:09.711830256 +0000 UTC m=+1976.052943743" watchObservedRunningTime="2026-11-03 20:00:09.712730086 +0000 UTC m=+1976.053843614" Nov 03 20:00:10.699221 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:10.699187 5233 generic.go:356] "Generic (PLEG): container finished" podID="c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" containerID="4a32091e416beaada586356f3ae95aee90c6c058ce58e4e77254142f844204fa" exitCode=0 Nov 03 20:00:10.699645 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:10.699273 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" event={"ID":"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa","Type":"ContainerDied","Data":"4a32091e416beaada586356f3ae95aee90c6c058ce58e4e77254142f844204fa"} Nov 03 20:00:11.855142 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.855118 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:11.938317 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.938283 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca\") pod \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " Nov 03 20:00:11.938507 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.938421 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca\") pod \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " Nov 03 20:00:11.938586 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.938513 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph\" (UniqueName: \"kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph\") pod \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\" (UID: \"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa\") " Nov 03 20:00:11.938779 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.938752 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca" pod "c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" (UID: "c1b2131e-8881-4023-b7c2-9b6d9d06ecaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:11.940378 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.940356 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca" pod "c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" (UID: "c1b2131e-8881-4023-b7c2-9b6d9d06ecaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:11.940641 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:11.940616 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph" pod "c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" (UID: "c1b2131e-8881-4023-b7c2-9b6d9d06ecaa"). InnerVolumeSpecName "kube-api-access-tjpph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:12.040132 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:12.040107 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:12.040132 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:12.040131 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpph\" (UniqueName: \"kubernetes.io/projected/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-kube-api-access-tjpph\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:12.040320 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:12.040141 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:12.712657 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:12.712618 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" event={"ID":"c1b2131e-8881-4023-b7c2-9b6d9d06ecaa","Type":"ContainerDied","Data":"fd972923593ac8eb8ae4e99ebdf734d70b649627dff15880e8a70809b8f0edb1"} Nov 03 20:00:12.712832 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:12.712677 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd972923593ac8eb8ae4e99ebdf734d70b649627dff15880e8a70809b8f0edb1" Nov 03 20:00:12.712832 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:12.712633 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn" Nov 03 20:00:16.829988 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:16.829941 5233 scope.go:122] "RemoveContainer" containerID="7305d03489095866b4503796150dd9fbb71f01ef2fc08cfb9b9a9426591486a3" Nov 03 20:00:17.745785 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:17.745757 5233 generic.go:356] "Generic (PLEG): container finished" podID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerID="de2590380764dcba5b3536dfa07536b6c875aa49c3acea0d409452a9b90562f6" exitCode=60 Nov 03 20:00:17.745963 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:17.745819 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" event={"ID":"b3dcb338-6ba6-4a7b-8b84-40befd7a216b","Type":"ContainerDied","Data":"de2590380764dcba5b3536dfa07536b6c875aa49c3acea0d409452a9b90562f6"} Nov 03 20:00:17.745963 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:17.745877 5233 scope.go:122] "RemoveContainer" containerID="7305d03489095866b4503796150dd9fbb71f01ef2fc08cfb9b9a9426591486a3" Nov 03 20:00:17.746306 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:17.746285 5233 scope.go:122] "RemoveContainer" containerID="de2590380764dcba5b3536dfa07536b6c875aa49c3acea0d409452a9b90562f6" Nov 03 20:00:17.746566 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:17.746543 5233 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-dtw7g_odh-ai-gateway-infra(b3dcb338-6ba6-4a7b-8b84-40befd7a216b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" Nov 03 20:00:18.774041 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:18.774006 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g"] Nov 03 20:00:18.917004 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:18.916964 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:19.009842 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.009808 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca\") pod \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " Nov 03 20:00:19.010089 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.009900 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2\" (UniqueName: \"kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2\") pod \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\" (UID: \"b3dcb338-6ba6-4a7b-8b84-40befd7a216b\") " Nov 03 20:00:19.010205 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.010163 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca" pod "b3dcb338-6ba6-4a7b-8b84-40befd7a216b" (UID: "b3dcb338-6ba6-4a7b-8b84-40befd7a216b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:19.010292 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.010265 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-maas-api-service-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.011901 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.011869 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2" pod "b3dcb338-6ba6-4a7b-8b84-40befd7a216b" (UID: "b3dcb338-6ba6-4a7b-8b84-40befd7a216b"). InnerVolumeSpecName "kube-api-access-q7xs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:19.111604 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.111533 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xs2\" (UniqueName: \"kubernetes.io/projected/b3dcb338-6ba6-4a7b-8b84-40befd7a216b-kube-api-access-q7xs2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.764724 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.764682 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" event={"ID":"b3dcb338-6ba6-4a7b-8b84-40befd7a216b","Type":"ContainerDied","Data":"a375cf92e0e42910a77f14cc2c9e3cd0c57f902c8fad552d55016cc5a5316040"} Nov 03 20:00:19.765023 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.764756 5233 scope.go:122] "RemoveContainer" containerID="de2590380764dcba5b3536dfa07536b6c875aa49c3acea0d409452a9b90562f6" Nov 03 20:00:19.765023 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.764938 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g" Nov 03 20:00:19.794856 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.794819 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g"] Nov 03 20:00:19.802367 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.802338 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-dtw7g"] Nov 03 20:00:19.841164 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.841132 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" path="/var/lib/kubelet/pods/b3dcb338-6ba6-4a7b-8b84-40befd7a216b/volumes" Nov 03 20:00:19.873815 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.873778 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c"] Nov 03 20:00:19.874355 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874332 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:19.874355 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874358 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874386 5233 cpu_manager.go:414] "containerMap: removing container" podUID="c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" containerName="revoke-keys" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874392 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" containerName="revoke-keys" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874417 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874426 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874527 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874540 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:19.874556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.874559 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" containerName="revoke-keys" Nov 03 20:00:19.879181 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.879162 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:19.885452 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.884776 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:19.885452 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.885167 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:00:19.888211 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.888187 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c"] Nov 03 20:00:19.919043 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.919010 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:19.919225 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.919197 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:19.919352 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:19.919237 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rmg\" (UniqueName: \"kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.020986 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.020886 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.021153 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.021017 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.021153 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.021052 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rmg\" (UniqueName: \"kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.021724 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.021701 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.023625 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.023590 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.033721 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.033693 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rmg\" (UniqueName: \"kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.196050 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.196017 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:20.478554 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.478528 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c"] Nov 03 20:00:20.480315 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:00:20.480291 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bf3a29_2e8f_4b22_8368_b27b38107ad2.slice/crio-1b922a8744e65c09c4ba0ab6cda12a87b032f4cb1ac37314762d25acaf273ebb WatchSource:0}: Error finding container 1b922a8744e65c09c4ba0ab6cda12a87b032f4cb1ac37314762d25acaf273ebb: Status 404 returned error can't find the container with id 1b922a8744e65c09c4ba0ab6cda12a87b032f4cb1ac37314762d25acaf273ebb Nov 03 20:00:20.774004 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:20.773912 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" event={"ID":"87bf3a29-2e8f-4b22-8368-b27b38107ad2","Type":"ContainerStarted","Data":"1b922a8744e65c09c4ba0ab6cda12a87b032f4cb1ac37314762d25acaf273ebb"} Nov 03 20:00:21.278380 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:21.278342 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:00:21.780894 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:21.780857 5233 generic.go:356] "Generic (PLEG): container finished" podID="87bf3a29-2e8f-4b22-8368-b27b38107ad2" containerID="e2dfeba3c59cc827ae3a4016913a5e03524b155d7f1b83e5c14d3fb93343f93f" exitCode=0 Nov 03 20:00:21.781110 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:21.780947 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" event={"ID":"87bf3a29-2e8f-4b22-8368-b27b38107ad2","Type":"ContainerDied","Data":"e2dfeba3c59cc827ae3a4016913a5e03524b155d7f1b83e5c14d3fb93343f93f"} Nov 03 20:00:22.753308 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:22.753275 5233 scope.go:122] "RemoveContainer" containerID="6d997dcf8d81a004ec20d726151e38e0346d3d346735d43e96396db392a2936a" Nov 03 20:00:22.945833 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:22.945808 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:23.061157 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.061069 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca\") pod \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " Nov 03 20:00:23.061379 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.061197 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca\") pod \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " Nov 03 20:00:23.061379 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.061249 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg\" (UniqueName: \"kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg\") pod \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\" (UID: \"87bf3a29-2e8f-4b22-8368-b27b38107ad2\") " Nov 03 20:00:23.061549 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.061524 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca" pod "87bf3a29-2e8f-4b22-8368-b27b38107ad2" (UID: "87bf3a29-2e8f-4b22-8368-b27b38107ad2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:23.063207 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.063180 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca" pod "87bf3a29-2e8f-4b22-8368-b27b38107ad2" (UID: "87bf3a29-2e8f-4b22-8368-b27b38107ad2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:23.063323 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.063299 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg" pod "87bf3a29-2e8f-4b22-8368-b27b38107ad2" (UID: "87bf3a29-2e8f-4b22-8368-b27b38107ad2"). InnerVolumeSpecName "kube-api-access-p5rmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:23.162758 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.162722 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:23.162758 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.162757 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rmg\" (UniqueName: \"kubernetes.io/projected/87bf3a29-2e8f-4b22-8368-b27b38107ad2-kube-api-access-p5rmg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:23.162758 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.162768 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bf3a29-2e8f-4b22-8368-b27b38107ad2-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:23.795786 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.795754 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" Nov 03 20:00:23.796266 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.795784 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c" event={"ID":"87bf3a29-2e8f-4b22-8368-b27b38107ad2","Type":"ContainerDied","Data":"1b922a8744e65c09c4ba0ab6cda12a87b032f4cb1ac37314762d25acaf273ebb"} Nov 03 20:00:23.796266 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:23.795862 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b922a8744e65c09c4ba0ab6cda12a87b032f4cb1ac37314762d25acaf273ebb" Nov 03 20:00:30.040214 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:30.040176 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g"] Nov 03 20:00:30.046670 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:30.046381 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh"] Nov 03 20:00:30.414786 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:30.414699 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c"] Nov 03 20:00:30.420766 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:30.420734 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-458393-pqg9c"] Nov 03 20:00:31.835717 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:31.835686 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87bf3a29-2e8f-4b22-8368-b27b38107ad2" path="/var/lib/kubelet/pods/87bf3a29-2e8f-4b22-8368-b27b38107ad2/volumes" Nov 03 20:00:35.041390 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:35.041338 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerName="payload-processing" containerID="cri-o://df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4" gracePeriod=25 Nov 03 20:00:35.046889 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:35.046860 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerName="payload-pre-processing" containerID="cri-o://d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837" gracePeriod=25 Nov 03 20:00:35.413846 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:35.413765 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:00:35.413846 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:35.413799 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:00:38.593273 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.593226 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t"] Nov 03 20:00:38.596760 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.596732 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vjc2t"] Nov 03 20:00:38.613809 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.613781 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g"] Nov 03 20:00:38.614520 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.614501 5233 cpu_manager.go:414] "containerMap: removing container" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:38.614578 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.614523 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:38.614578 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.614542 5233 cpu_manager.go:414] "containerMap: removing container" podUID="87bf3a29-2e8f-4b22-8368-b27b38107ad2" containerName="revoke-keys" Nov 03 20:00:38.614578 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.614550 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87bf3a29-2e8f-4b22-8368-b27b38107ad2" containerName="revoke-keys" Nov 03 20:00:38.614686 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.614644 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dcb338-6ba6-4a7b-8b84-40befd7a216b" containerName="cleanup" Nov 03 20:00:38.614686 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.614656 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="87bf3a29-2e8f-4b22-8368-b27b38107ad2" containerName="revoke-keys" Nov 03 20:00:38.618402 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.618384 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.622795 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.622770 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:00:38.622932 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.622845 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:38.626888 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.626862 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g"] Nov 03 20:00:38.727250 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.727213 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.727444 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.727396 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.727534 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.727507 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hqh\" (UniqueName: \"kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.828674 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.828637 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.828897 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.828712 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hqh\" (UniqueName: \"kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.828897 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.828770 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.829505 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.829476 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.831224 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.831194 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.841264 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.841239 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hqh\" (UniqueName: \"kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh\") pod \"maas-api-revoke-sub-3d4156f1-fdr7g\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:38.937162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:38.937081 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:39.218640 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:39.218615 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g"] Nov 03 20:00:39.221202 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:00:39.221165 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cd0878_af6f_4f01_a729_5e1da1301d4d.slice/crio-e1e95a4737c8051727159501038d9c4bb2771f14936b4ec7ed1db511e1c235ee WatchSource:0}: Error finding container e1e95a4737c8051727159501038d9c4bb2771f14936b4ec7ed1db511e1c235ee: Status 404 returned error can't find the container with id e1e95a4737c8051727159501038d9c4bb2771f14936b4ec7ed1db511e1c235ee Nov 03 20:00:39.835736 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:39.835665 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd882be8-2380-4be8-b135-13699c3331ff" path="/var/lib/kubelet/pods/dd882be8-2380-4be8-b135-13699c3331ff/volumes" Nov 03 20:00:39.887847 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:39.887809 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" event={"ID":"00cd0878-af6f-4f01-a729-5e1da1301d4d","Type":"ContainerStarted","Data":"c3c83de0ea53e77dc10ce2d356a376a6a04e60b9e583b1cf5fedadf91f83fcb2"} Nov 03 20:00:39.888042 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:39.887873 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" event={"ID":"00cd0878-af6f-4f01-a729-5e1da1301d4d","Type":"ContainerStarted","Data":"e1e95a4737c8051727159501038d9c4bb2771f14936b4ec7ed1db511e1c235ee"} Nov 03 20:00:39.908935 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:39.908881 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" podStartSLOduration=1.908863727 podStartE2EDuration="1.908863727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:39.907085663 +0000 UTC m=+2006.248199162" watchObservedRunningTime="2026-11-03 20:00:39.908863727 +0000 UTC m=+2006.249977213" Nov 03 20:00:40.895210 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:40.895181 5233 generic.go:356] "Generic (PLEG): container finished" podID="00cd0878-af6f-4f01-a729-5e1da1301d4d" containerID="c3c83de0ea53e77dc10ce2d356a376a6a04e60b9e583b1cf5fedadf91f83fcb2" exitCode=0 Nov 03 20:00:40.895662 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:40.895268 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" event={"ID":"00cd0878-af6f-4f01-a729-5e1da1301d4d","Type":"ContainerDied","Data":"c3c83de0ea53e77dc10ce2d356a376a6a04e60b9e583b1cf5fedadf91f83fcb2"} Nov 03 20:00:42.053376 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.053344 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:42.164884 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.164795 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh\" (UniqueName: \"kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh\") pod \"00cd0878-af6f-4f01-a729-5e1da1301d4d\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " Nov 03 20:00:42.164884 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.164850 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca\") pod \"00cd0878-af6f-4f01-a729-5e1da1301d4d\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " Nov 03 20:00:42.165160 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.165021 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca\") pod \"00cd0878-af6f-4f01-a729-5e1da1301d4d\" (UID: \"00cd0878-af6f-4f01-a729-5e1da1301d4d\") " Nov 03 20:00:42.165362 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.165338 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca" pod "00cd0878-af6f-4f01-a729-5e1da1301d4d" (UID: "00cd0878-af6f-4f01-a729-5e1da1301d4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:42.166954 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.166930 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca" pod "00cd0878-af6f-4f01-a729-5e1da1301d4d" (UID: "00cd0878-af6f-4f01-a729-5e1da1301d4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:42.167025 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.166933 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh" pod "00cd0878-af6f-4f01-a729-5e1da1301d4d" (UID: "00cd0878-af6f-4f01-a729-5e1da1301d4d"). InnerVolumeSpecName "kube-api-access-w7hqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:42.266667 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.266634 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.266667 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.266661 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7hqh\" (UniqueName: \"kubernetes.io/projected/00cd0878-af6f-4f01-a729-5e1da1301d4d-kube-api-access-w7hqh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.266667 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.266672 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cd0878-af6f-4f01-a729-5e1da1301d4d-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.906698 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.906666 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" Nov 03 20:00:42.906888 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.906695 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g" event={"ID":"00cd0878-af6f-4f01-a729-5e1da1301d4d","Type":"ContainerDied","Data":"e1e95a4737c8051727159501038d9c4bb2771f14936b4ec7ed1db511e1c235ee"} Nov 03 20:00:42.906888 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:42.906745 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e95a4737c8051727159501038d9c4bb2771f14936b4ec7ed1db511e1c235ee" Nov 03 20:00:45.413771 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:45.413733 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:00:45.414194 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:45.413801 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:00:51.275883 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:51.275838 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:00:54.102721 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.102646 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g"] Nov 03 20:00:54.103124 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.102838 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" podUID="75643011-dadc-414a-8aeb-0629a61ae5ca" containerName="manager" containerID="cri-o://3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a" gracePeriod=10 Nov 03 20:00:54.372127 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.372098 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 20:00:54.495067 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.495037 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume\") pod \"75643011-dadc-414a-8aeb-0629a61ae5ca\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " Nov 03 20:00:54.495231 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.495097 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq\" (UniqueName: \"kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq\") pod \"75643011-dadc-414a-8aeb-0629a61ae5ca\" (UID: \"75643011-dadc-414a-8aeb-0629a61ae5ca\") " Nov 03 20:00:54.495470 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.495449 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume" pod "75643011-dadc-414a-8aeb-0629a61ae5ca" (UID: "75643011-dadc-414a-8aeb-0629a61ae5ca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:54.497225 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.497195 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq" pod "75643011-dadc-414a-8aeb-0629a61ae5ca" (UID: "75643011-dadc-414a-8aeb-0629a61ae5ca"). InnerVolumeSpecName "kube-api-access-79fmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:54.596426 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.596394 5233 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75643011-dadc-414a-8aeb-0629a61ae5ca-extensions-socket-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:54.596426 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.596422 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79fmq\" (UniqueName: \"kubernetes.io/projected/75643011-dadc-414a-8aeb-0629a61ae5ca-kube-api-access-79fmq\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:54.974922 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.974890 5233 generic.go:356] "Generic (PLEG): container finished" podID="75643011-dadc-414a-8aeb-0629a61ae5ca" containerID="3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a" exitCode=0 Nov 03 20:00:54.975114 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.974936 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" Nov 03 20:00:54.975114 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.974986 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" event={"ID":"75643011-dadc-414a-8aeb-0629a61ae5ca","Type":"ContainerDied","Data":"3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a"} Nov 03 20:00:54.975114 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.975028 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g" event={"ID":"75643011-dadc-414a-8aeb-0629a61ae5ca","Type":"ContainerDied","Data":"40a216af349b99ff41b706adc356137127c3b47588d14f2cffd5ffb82570706b"} Nov 03 20:00:54.975114 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.975048 5233 scope.go:122] "RemoveContainer" containerID="3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a" Nov 03 20:00:54.987393 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.987373 5233 scope.go:122] "RemoveContainer" containerID="3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a" Nov 03 20:00:54.987655 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:54.987636 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a\": container with ID starting with 3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a not found: ID does not exist" containerID="3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a" Nov 03 20:00:54.987715 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.987664 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a"} err="failed to get container status \"3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a\": rpc error: code = NotFound desc = could not find container \"3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a\": container with ID starting with 3f9ddb98212383d00945ce9d011f6fb664e98d28b5f16a55948512074c2a695a not found: ID does not exist" Nov 03 20:00:54.999231 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:54.999204 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g"] Nov 03 20:00:55.002894 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.002873 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lcb4g"] Nov 03 20:00:55.296786 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.296762 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 20:00:55.329305 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.329282 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 20:00:55.404699 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.404659 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb\" (UniqueName: \"kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb\") pod \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " Nov 03 20:00:55.404847 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.404743 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume\") pod \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\" (UID: \"128dd3cc-0329-4d7f-b9c0-58a33a03dfad\") " Nov 03 20:00:55.404847 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.404808 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72\" (UniqueName: \"kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72\") pod \"9e70130e-ba79-49e5-8d57-709fba1d75d7\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " Nov 03 20:00:55.404847 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.404833 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume\") pod \"9e70130e-ba79-49e5-8d57-709fba1d75d7\" (UID: \"9e70130e-ba79-49e5-8d57-709fba1d75d7\") " Nov 03 20:00:55.405297 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.405271 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume" pod "128dd3cc-0329-4d7f-b9c0-58a33a03dfad" (UID: "128dd3cc-0329-4d7f-b9c0-58a33a03dfad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:55.405395 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.405279 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume" pod "9e70130e-ba79-49e5-8d57-709fba1d75d7" (UID: "9e70130e-ba79-49e5-8d57-709fba1d75d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:55.406756 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.406735 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb" pod "128dd3cc-0329-4d7f-b9c0-58a33a03dfad" (UID: "128dd3cc-0329-4d7f-b9c0-58a33a03dfad"). InnerVolumeSpecName "kube-api-access-25clb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:55.406840 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.406822 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72" pod "9e70130e-ba79-49e5-8d57-709fba1d75d7" (UID: "9e70130e-ba79-49e5-8d57-709fba1d75d7"). InnerVolumeSpecName "kube-api-access-v4n72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:55.506514 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.506441 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4n72\" (UniqueName: \"kubernetes.io/projected/9e70130e-ba79-49e5-8d57-709fba1d75d7-kube-api-access-v4n72\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:55.506514 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.506473 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e70130e-ba79-49e5-8d57-709fba1d75d7-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:55.506514 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.506489 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25clb\" (UniqueName: \"kubernetes.io/projected/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-kube-api-access-25clb\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:55.506514 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.506504 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dd3cc-0329-4d7f-b9c0-58a33a03dfad-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:55.836170 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.836095 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75643011-dadc-414a-8aeb-0629a61ae5ca" path="/var/lib/kubelet/pods/75643011-dadc-414a-8aeb-0629a61ae5ca/volumes" Nov 03 20:00:55.981510 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.981477 5233 generic.go:356] "Generic (PLEG): container finished" podID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerID="df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4" exitCode=0 Nov 03 20:00:55.981703 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.981523 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" Nov 03 20:00:55.981703 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.981555 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" event={"ID":"128dd3cc-0329-4d7f-b9c0-58a33a03dfad","Type":"ContainerDied","Data":"df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4"} Nov 03 20:00:55.981703 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.981607 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g" event={"ID":"128dd3cc-0329-4d7f-b9c0-58a33a03dfad","Type":"ContainerDied","Data":"f4e2b8d02303ed5ec713005e49276d8619ae83e7bd2544d9cf9dc640c6b08486"} Nov 03 20:00:55.981703 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.981628 5233 scope.go:122] "RemoveContainer" containerID="df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4" Nov 03 20:00:55.982908 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.982888 5233 generic.go:356] "Generic (PLEG): container finished" podID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerID="d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837" exitCode=0 Nov 03 20:00:55.983008 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.982931 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" Nov 03 20:00:55.983080 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.983010 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" event={"ID":"9e70130e-ba79-49e5-8d57-709fba1d75d7","Type":"ContainerDied","Data":"d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837"} Nov 03 20:00:55.983080 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.983058 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh" event={"ID":"9e70130e-ba79-49e5-8d57-709fba1d75d7","Type":"ContainerDied","Data":"4c0b43fbc0f45a337ce77ff6f27119a955459772a8f73e3feaf57e5372d2414f"} Nov 03 20:00:55.991273 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.991257 5233 scope.go:122] "RemoveContainer" containerID="df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4" Nov 03 20:00:55.991532 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:55.991513 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4\": container with ID starting with df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4 not found: ID does not exist" containerID="df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4" Nov 03 20:00:55.991598 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.991539 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4"} err="failed to get container status \"df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4\": rpc error: code = NotFound desc = could not find container \"df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4\": container with ID starting with df8b7f746ea2735124a97ea245ddb76fdbe7c9d62360701ae8c4f0375f35d9c4 not found: ID does not exist" Nov 03 20:00:55.991598 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:55.991555 5233 scope.go:122] "RemoveContainer" containerID="d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837" Nov 03 20:00:56.000254 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:56.000232 5233 scope.go:122] "RemoveContainer" containerID="d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837" Nov 03 20:00:56.000487 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:00:56.000469 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837\": container with ID starting with d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837 not found: ID does not exist" containerID="d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837" Nov 03 20:00:56.000532 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:56.000493 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837"} err="failed to get container status \"d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837\": rpc error: code = NotFound desc = could not find container \"d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837\": container with ID starting with d53c650234e16d087133951e40473156308057b685c31081bf465a4c0e4e6837 not found: ID does not exist" Nov 03 20:00:56.001537 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:56.001519 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g"] Nov 03 20:00:56.005403 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:56.005379 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-458393-56ccb47d7zd96g"] Nov 03 20:00:56.020701 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:56.020679 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh"] Nov 03 20:00:56.024182 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:56.024157 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-458393-59868hjlkh"] Nov 03 20:00:57.835129 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:57.835092 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" path="/var/lib/kubelet/pods/128dd3cc-0329-4d7f-b9c0-58a33a03dfad/volumes" Nov 03 20:00:57.835652 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:00:57.835609 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" path="/var/lib/kubelet/pods/9e70130e-ba79-49e5-8d57-709fba1d75d7/volumes" Nov 03 20:01:21.276253 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:01:21.276219 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:01:51.276723 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:01:51.276686 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:01:55.149903 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.149858 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp"] Nov 03 20:01:55.150526 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150435 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerName="payload-pre-processing" Nov 03 20:01:55.150526 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150457 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerName="payload-pre-processing" Nov 03 20:01:55.150526 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150512 5233 cpu_manager.go:414] "containerMap: removing container" podUID="00cd0878-af6f-4f01-a729-5e1da1301d4d" containerName="revoke-keys" Nov 03 20:01:55.150526 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150520 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00cd0878-af6f-4f01-a729-5e1da1301d4d" containerName="revoke-keys" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150559 5233 cpu_manager.go:414] "containerMap: removing container" podUID="75643011-dadc-414a-8aeb-0629a61ae5ca" containerName="manager" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150567 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75643011-dadc-414a-8aeb-0629a61ae5ca" containerName="manager" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150578 5233 cpu_manager.go:414] "containerMap: removing container" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerName="payload-processing" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150586 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerName="payload-processing" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150684 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="00cd0878-af6f-4f01-a729-5e1da1301d4d" containerName="revoke-keys" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150694 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="75643011-dadc-414a-8aeb-0629a61ae5ca" containerName="manager" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150701 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="128dd3cc-0329-4d7f-b9c0-58a33a03dfad" containerName="payload-processing" Nov 03 20:01:55.150746 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.150709 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e70130e-ba79-49e5-8d57-709fba1d75d7" containerName="payload-pre-processing" Nov 03 20:01:55.153849 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.153829 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.159628 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.159607 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-brtl6\"" Nov 03 20:01:55.173644 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.173621 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp"] Nov 03 20:01:55.278856 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.278819 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/475ce0c0-4e75-4751-bdff-887b02e6c593-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7dsgp\" (UID: \"475ce0c0-4e75-4751-bdff-887b02e6c593\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.279033 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.278957 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzhrp\" (UniqueName: \"kubernetes.io/projected/475ce0c0-4e75-4751-bdff-887b02e6c593-kube-api-access-kzhrp\") pod \"kuadrant-operator-controller-manager-678d87766f-7dsgp\" (UID: \"475ce0c0-4e75-4751-bdff-887b02e6c593\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.379939 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.379904 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/475ce0c0-4e75-4751-bdff-887b02e6c593-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7dsgp\" (UID: \"475ce0c0-4e75-4751-bdff-887b02e6c593\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.380122 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.379995 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzhrp\" (UniqueName: \"kubernetes.io/projected/475ce0c0-4e75-4751-bdff-887b02e6c593-kube-api-access-kzhrp\") pod \"kuadrant-operator-controller-manager-678d87766f-7dsgp\" (UID: \"475ce0c0-4e75-4751-bdff-887b02e6c593\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.380305 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.380284 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/475ce0c0-4e75-4751-bdff-887b02e6c593-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7dsgp\" (UID: \"475ce0c0-4e75-4751-bdff-887b02e6c593\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.396667 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.396641 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzhrp\" (UniqueName: \"kubernetes.io/projected/475ce0c0-4e75-4751-bdff-887b02e6c593-kube-api-access-kzhrp\") pod \"kuadrant-operator-controller-manager-678d87766f-7dsgp\" (UID: \"475ce0c0-4e75-4751-bdff-887b02e6c593\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.467578 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.467556 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:55.760750 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:55.760716 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp"] Nov 03 20:01:55.762056 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:01:55.762023 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475ce0c0_4e75_4751_bdff_887b02e6c593.slice/crio-ea1e7b397f9e1759783a60f5a6a0a5c3f1eabc1fb10bdd44ca982855b8a60adf WatchSource:0}: Error finding container ea1e7b397f9e1759783a60f5a6a0a5c3f1eabc1fb10bdd44ca982855b8a60adf: Status 404 returned error can't find the container with id ea1e7b397f9e1759783a60f5a6a0a5c3f1eabc1fb10bdd44ca982855b8a60adf Nov 03 20:01:56.313741 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:56.313699 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" event={"ID":"475ce0c0-4e75-4751-bdff-887b02e6c593","Type":"ContainerStarted","Data":"ae20e4fe206afd8857bd38c5bd73472e252056dd560813968b8bda69a955d7ba"} Nov 03 20:01:56.314188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:56.313771 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:01:56.314188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:56.313798 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" event={"ID":"475ce0c0-4e75-4751-bdff-887b02e6c593","Type":"ContainerStarted","Data":"ea1e7b397f9e1759783a60f5a6a0a5c3f1eabc1fb10bdd44ca982855b8a60adf"} Nov 03 20:01:56.348964 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:01:56.348906 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" podStartSLOduration=1.348887098 podStartE2EDuration="1.348887098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:56.345961332 +0000 UTC m=+2082.687074818" watchObservedRunningTime="2026-11-03 20:01:56.348887098 +0000 UTC m=+2082.690000586" Nov 03 20:02:07.322549 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:07.322518 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7dsgp" Nov 03 20:02:20.044283 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.044247 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4"] Nov 03 20:02:20.048031 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.048015 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.056159 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.056132 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:20.056313 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.056227 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:02:20.057423 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.057397 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4"] Nov 03 20:02:20.087700 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.087650 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.087898 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.087845 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.088242 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.087945 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9rs\" (UniqueName: \"kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.189190 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.189154 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.189351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.189230 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.189351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.189270 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9rs\" (UniqueName: \"kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.189775 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.189754 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.191530 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.191503 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.201900 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.201875 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9rs\" (UniqueName: \"kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs\") pod \"maas-api-revoke-sub-d576e68b-qbkt4\" (UID: \"244893ed-5770-407b-8491-007710541082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.362756 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.362672 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:20.642224 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:20.642197 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4"] Nov 03 20:02:20.644215 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:02:20.644183 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244893ed_5770_407b_8491_007710541082.slice/crio-b23d733c04bd1fd02759d53270ff56db89cac6a8bc4a38e55069a44675506a3b WatchSource:0}: Error finding container b23d733c04bd1fd02759d53270ff56db89cac6a8bc4a38e55069a44675506a3b: Status 404 returned error can't find the container with id b23d733c04bd1fd02759d53270ff56db89cac6a8bc4a38e55069a44675506a3b Nov 03 20:02:21.276748 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:02:21.276716 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:02:21.620183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:21.620096 5233 generic.go:356] "Generic (PLEG): container finished" podID="244893ed-5770-407b-8491-007710541082" containerID="a4c7731681f05159d4c601c1aaef58d75dc1c939cc06b2574659b51112a09212" exitCode=0 Nov 03 20:02:21.620351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:21.620192 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" event={"ID":"244893ed-5770-407b-8491-007710541082","Type":"ContainerDied","Data":"a4c7731681f05159d4c601c1aaef58d75dc1c939cc06b2574659b51112a09212"} Nov 03 20:02:21.620351 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:21.620257 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" event={"ID":"244893ed-5770-407b-8491-007710541082","Type":"ContainerStarted","Data":"b23d733c04bd1fd02759d53270ff56db89cac6a8bc4a38e55069a44675506a3b"} Nov 03 20:02:22.762648 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.762625 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:22.815794 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.815753 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs\" (UniqueName: \"kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs\") pod \"244893ed-5770-407b-8491-007710541082\" (UID: \"244893ed-5770-407b-8491-007710541082\") " Nov 03 20:02:22.816017 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.815837 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca\") pod \"244893ed-5770-407b-8491-007710541082\" (UID: \"244893ed-5770-407b-8491-007710541082\") " Nov 03 20:02:22.816086 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.816021 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca\") pod \"244893ed-5770-407b-8491-007710541082\" (UID: \"244893ed-5770-407b-8491-007710541082\") " Nov 03 20:02:22.816300 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.816268 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca" pod "244893ed-5770-407b-8491-007710541082" (UID: "244893ed-5770-407b-8491-007710541082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:22.816437 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.816388 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244893ed-5770-407b-8491-007710541082-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:22.818100 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.818078 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs" pod "244893ed-5770-407b-8491-007710541082" (UID: "244893ed-5770-407b-8491-007710541082"). InnerVolumeSpecName "kube-api-access-bs9rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:22.818730 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.818706 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca" pod "244893ed-5770-407b-8491-007710541082" (UID: "244893ed-5770-407b-8491-007710541082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:22.917737 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.917671 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244893ed-5770-407b-8491-007710541082-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:22.917737 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:22.917697 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9rs\" (UniqueName: \"kubernetes.io/projected/244893ed-5770-407b-8491-007710541082-kube-api-access-bs9rs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:23.631853 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:23.631779 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" Nov 03 20:02:23.631853 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:23.631816 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4" event={"ID":"244893ed-5770-407b-8491-007710541082","Type":"ContainerDied","Data":"b23d733c04bd1fd02759d53270ff56db89cac6a8bc4a38e55069a44675506a3b"} Nov 03 20:02:23.632081 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:23.631866 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23d733c04bd1fd02759d53270ff56db89cac6a8bc4a38e55069a44675506a3b" Nov 03 20:02:35.398607 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:35.398082 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:02:35.398607 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:35.398154 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:35.594097 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:35.594058 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:02:35.594387 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:35.594352 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:35.622378 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:35.622340 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:02:35.622646 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:35.622596 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:36.109094 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:36.109056 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:02:36.109423 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:36.109390 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:39.076433 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.076396 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69"] Nov 03 20:02:39.076868 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.076833 5233 cpu_manager.go:414] "containerMap: removing container" podUID="244893ed-5770-407b-8491-007710541082" containerName="revoke-keys" Nov 03 20:02:39.076868 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.076843 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244893ed-5770-407b-8491-007710541082" containerName="revoke-keys" Nov 03 20:02:39.076960 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.076920 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="244893ed-5770-407b-8491-007710541082" containerName="revoke-keys" Nov 03 20:02:39.080175 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.080157 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.088124 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.088099 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:39.089804 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.089782 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:02:39.091693 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.091673 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69"] Nov 03 20:02:39.245287 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.245254 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.245483 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.245407 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.245483 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.245435 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmg7d\" (UniqueName: \"kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.346638 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.346555 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmg7d\" (UniqueName: \"kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.346638 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.346623 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.346851 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.346726 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.347454 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.347433 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.349252 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.349229 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.359703 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.359679 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmg7d\" (UniqueName: \"kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d\") pod \"maas-api-revoke-sub-d2bc93bc-jvl69\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.394478 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.394454 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:39.685246 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.685215 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69"] Nov 03 20:02:39.687287 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:02:39.687253 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7adeb9_326a_4193_8bac_741fe3b6ec0c.slice/crio-9b65466f5e3d19f6147776c9a3c45c6734db876d363c0c8d8df7dd4f8d880673 WatchSource:0}: Error finding container 9b65466f5e3d19f6147776c9a3c45c6734db876d363c0c8d8df7dd4f8d880673: Status 404 returned error can't find the container with id 9b65466f5e3d19f6147776c9a3c45c6734db876d363c0c8d8df7dd4f8d880673 Nov 03 20:02:39.769134 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:39.769100 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" event={"ID":"9c7adeb9-326a-4193-8bac-741fe3b6ec0c","Type":"ContainerStarted","Data":"9b65466f5e3d19f6147776c9a3c45c6734db876d363c0c8d8df7dd4f8d880673"} Nov 03 20:02:40.776116 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:40.776080 5233 generic.go:356] "Generic (PLEG): container finished" podID="9c7adeb9-326a-4193-8bac-741fe3b6ec0c" containerID="4ca0fe0b449e6cab67105ca56939a28a0d743a83e8987f1f9570b1d54f742f74" exitCode=0 Nov 03 20:02:40.776491 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:40.776119 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" event={"ID":"9c7adeb9-326a-4193-8bac-741fe3b6ec0c","Type":"ContainerDied","Data":"4ca0fe0b449e6cab67105ca56939a28a0d743a83e8987f1f9570b1d54f742f74"} Nov 03 20:02:41.915054 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.915033 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:41.971133 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.971095 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca\") pod \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " Nov 03 20:02:41.971323 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.971163 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca\") pod \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " Nov 03 20:02:41.971323 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.971266 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d\" (UniqueName: \"kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d\") pod \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\" (UID: \"9c7adeb9-326a-4193-8bac-741fe3b6ec0c\") " Nov 03 20:02:41.971492 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.971469 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca" pod "9c7adeb9-326a-4193-8bac-741fe3b6ec0c" (UID: "9c7adeb9-326a-4193-8bac-741fe3b6ec0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:41.971655 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.971637 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:41.973232 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.973211 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca" pod "9c7adeb9-326a-4193-8bac-741fe3b6ec0c" (UID: "9c7adeb9-326a-4193-8bac-741fe3b6ec0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:41.973339 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:41.973291 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d" pod "9c7adeb9-326a-4193-8bac-741fe3b6ec0c" (UID: "9c7adeb9-326a-4193-8bac-741fe3b6ec0c"). InnerVolumeSpecName "kube-api-access-qmg7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:42.072285 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:42.072214 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:42.072285 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:42.072242 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmg7d\" (UniqueName: \"kubernetes.io/projected/9c7adeb9-326a-4193-8bac-741fe3b6ec0c-kube-api-access-qmg7d\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:42.791718 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:42.791686 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" Nov 03 20:02:42.791718 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:42.791710 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69" event={"ID":"9c7adeb9-326a-4193-8bac-741fe3b6ec0c","Type":"ContainerDied","Data":"9b65466f5e3d19f6147776c9a3c45c6734db876d363c0c8d8df7dd4f8d880673"} Nov 03 20:02:42.791945 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:42.791756 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b65466f5e3d19f6147776c9a3c45c6734db876d363c0c8d8df7dd4f8d880673" Nov 03 20:02:51.277991 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:02:51.277946 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:02:56.225285 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.225248 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q"] Nov 03 20:02:56.225782 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.225766 5233 cpu_manager.go:414] "containerMap: removing container" podUID="9c7adeb9-326a-4193-8bac-741fe3b6ec0c" containerName="revoke-keys" Nov 03 20:02:56.225848 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.225785 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7adeb9-326a-4193-8bac-741fe3b6ec0c" containerName="revoke-keys" Nov 03 20:02:56.225954 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.225941 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7adeb9-326a-4193-8bac-741fe3b6ec0c" containerName="revoke-keys" Nov 03 20:02:56.229600 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.229584 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.247647 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.247609 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ddb898b9-dockercfg-4k449\"" Nov 03 20:02:56.247815 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.247680 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fcdf83\"" Nov 03 20:02:56.250488 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.250465 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q"] Nov 03 20:02:56.279417 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.279388 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt"] Nov 03 20:02:56.283150 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.283125 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.295882 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.295858 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt"] Nov 03 20:02:56.312404 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.312380 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.312521 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.312502 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zgh\" (UniqueName: \"kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh\") pod \"payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.413661 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.413628 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.413862 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.413676 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfjxg\" (UniqueName: \"kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg\") pod \"payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.413862 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.413802 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zgh\" (UniqueName: \"kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh\") pod \"payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.414015 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.413923 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.414580 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.414562 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.427921 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.427887 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zgh\" (UniqueName: \"kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh\") pod \"payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.515346 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.515262 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.515346 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.515312 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfjxg\" (UniqueName: \"kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg\") pod \"payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.516031 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.516009 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.528544 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.528512 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfjxg\" (UniqueName: \"kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg\") pod \"payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.543814 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.543790 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:56.599098 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.598133 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:56.829486 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.829453 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q"] Nov 03 20:02:56.830343 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:02:56.830314 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d7b260_8d86_4454_b58b_094e167875cc.slice/crio-2482eb1a6594a77d31fa2cbfac3129d6602061f9ed3fd2a61cc43275d6f98951 WatchSource:0}: Error finding container 2482eb1a6594a77d31fa2cbfac3129d6602061f9ed3fd2a61cc43275d6f98951: Status 404 returned error can't find the container with id 2482eb1a6594a77d31fa2cbfac3129d6602061f9ed3fd2a61cc43275d6f98951 Nov 03 20:02:56.871472 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.871449 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" event={"ID":"a7d7b260-8d86-4454-b58b-094e167875cc","Type":"ContainerStarted","Data":"2482eb1a6594a77d31fa2cbfac3129d6602061f9ed3fd2a61cc43275d6f98951"} Nov 03 20:02:56.912519 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:56.912497 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt"] Nov 03 20:02:56.914039 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:02:56.914011 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f46cbf1_0500_4132_acb9_11ed593201ae.slice/crio-13b125188055bebc3ca5977107fa8608333740d3d949d45a6adad776e7017984 WatchSource:0}: Error finding container 13b125188055bebc3ca5977107fa8608333740d3d949d45a6adad776e7017984: Status 404 returned error can't find the container with id 13b125188055bebc3ca5977107fa8608333740d3d949d45a6adad776e7017984 Nov 03 20:02:57.884134 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.881702 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" event={"ID":"a7d7b260-8d86-4454-b58b-094e167875cc","Type":"ContainerStarted","Data":"26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6"} Nov 03 20:02:57.884134 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.881810 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:02:57.885410 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.885373 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" event={"ID":"2f46cbf1-0500-4132-acb9-11ed593201ae","Type":"ContainerStarted","Data":"10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16"} Nov 03 20:02:57.885626 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.885446 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" event={"ID":"2f46cbf1-0500-4132-acb9-11ed593201ae","Type":"ContainerStarted","Data":"13b125188055bebc3ca5977107fa8608333740d3d949d45a6adad776e7017984"} Nov 03 20:02:57.886427 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.886316 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:02:57.925057 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.924990 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" podStartSLOduration=1.924957756 podStartE2EDuration="1.924957756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:57.922466039 +0000 UTC m=+2144.263579525" watchObservedRunningTime="2026-11-03 20:02:57.924957756 +0000 UTC m=+2144.266071241" Nov 03 20:02:57.947839 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:02:57.947792 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" podStartSLOduration=1.947779138 podStartE2EDuration="1.947779138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:57.947694512 +0000 UTC m=+2144.288807996" watchObservedRunningTime="2026-11-03 20:02:57.947779138 +0000 UTC m=+2144.288892644" Nov 03 20:03:08.895940 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:08.895908 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:03:09.901817 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:09.901782 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:03:21.275794 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:03:21.275755 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:03:42.431143 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.431109 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8"] Nov 03 20:03:42.435004 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.434966 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.444181 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.444152 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:42.444315 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.444189 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:03:42.446148 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.446123 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8"] Nov 03 20:03:42.600677 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.600642 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.600894 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.600687 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwph\" (UniqueName: \"kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.600894 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.600748 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.702003 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.701892 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.702003 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.701959 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwph\" (UniqueName: \"kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.702246 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.702037 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.702639 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.702611 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.704289 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.704269 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.718261 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.718236 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwph\" (UniqueName: \"kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph\") pod \"maas-api-revoke-sub-712657a1-bhbc8\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:42.749368 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:42.749344 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:43.037772 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:43.037743 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8"] Nov 03 20:03:43.042690 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:03:43.042656 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3238700_fb6a_421a_a36f_8dd0ab8aef26.slice/crio-1e452e59817a7c8fb00ecf7926fd18719ca0186d1b55e8c468814eb5952bc958 WatchSource:0}: Error finding container 1e452e59817a7c8fb00ecf7926fd18719ca0186d1b55e8c468814eb5952bc958: Status 404 returned error can't find the container with id 1e452e59817a7c8fb00ecf7926fd18719ca0186d1b55e8c468814eb5952bc958 Nov 03 20:03:43.183318 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:43.183282 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" event={"ID":"d3238700-fb6a-421a-a36f-8dd0ab8aef26","Type":"ContainerStarted","Data":"1e452e59817a7c8fb00ecf7926fd18719ca0186d1b55e8c468814eb5952bc958"} Nov 03 20:03:44.197682 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:44.197591 5233 generic.go:356] "Generic (PLEG): container finished" podID="d3238700-fb6a-421a-a36f-8dd0ab8aef26" containerID="b59a4fd2ea60defa8147e4925a7c61d27af9f412b7af400ec5fd4ab1a67068eb" exitCode=0 Nov 03 20:03:44.197682 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:44.197629 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" event={"ID":"d3238700-fb6a-421a-a36f-8dd0ab8aef26","Type":"ContainerDied","Data":"b59a4fd2ea60defa8147e4925a7c61d27af9f412b7af400ec5fd4ab1a67068eb"} Nov 03 20:03:45.358004 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.357945 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:45.532988 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.532951 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph\" (UniqueName: \"kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph\") pod \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " Nov 03 20:03:45.533166 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.533103 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca\") pod \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " Nov 03 20:03:45.533166 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.533137 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca\") pod \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\" (UID: \"d3238700-fb6a-421a-a36f-8dd0ab8aef26\") " Nov 03 20:03:45.533507 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.533479 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca" pod "d3238700-fb6a-421a-a36f-8dd0ab8aef26" (UID: "d3238700-fb6a-421a-a36f-8dd0ab8aef26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:45.535182 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.535151 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca" pod "d3238700-fb6a-421a-a36f-8dd0ab8aef26" (UID: "d3238700-fb6a-421a-a36f-8dd0ab8aef26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:45.535182 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.535151 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph" pod "d3238700-fb6a-421a-a36f-8dd0ab8aef26" (UID: "d3238700-fb6a-421a-a36f-8dd0ab8aef26"). InnerVolumeSpecName "kube-api-access-9bwph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:45.634257 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.634225 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwph\" (UniqueName: \"kubernetes.io/projected/d3238700-fb6a-421a-a36f-8dd0ab8aef26-kube-api-access-9bwph\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:45.634257 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.634252 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:45.634257 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:45.634262 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3238700-fb6a-421a-a36f-8dd0ab8aef26-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:46.212039 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:46.211953 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" Nov 03 20:03:46.212039 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:46.212006 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8" event={"ID":"d3238700-fb6a-421a-a36f-8dd0ab8aef26","Type":"ContainerDied","Data":"1e452e59817a7c8fb00ecf7926fd18719ca0186d1b55e8c468814eb5952bc958"} Nov 03 20:03:46.212233 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:46.212053 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e452e59817a7c8fb00ecf7926fd18719ca0186d1b55e8c468814eb5952bc958" Nov 03 20:03:50.023575 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:50.023538 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx"] Nov 03 20:03:50.035164 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:50.029004 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hqhpx"] Nov 03 20:03:51.276086 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:03:51.276046 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:03:51.835028 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:51.834966 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c687550b-4991-4ae6-9942-56425726b5fe" path="/var/lib/kubelet/pods/c687550b-4991-4ae6-9942-56425726b5fe/volumes" Nov 03 20:03:53.331903 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.331862 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q"] Nov 03 20:03:53.332433 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.332340 5233 cpu_manager.go:414] "containerMap: removing container" podUID="d3238700-fb6a-421a-a36f-8dd0ab8aef26" containerName="revoke-keys" Nov 03 20:03:53.332433 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.332353 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3238700-fb6a-421a-a36f-8dd0ab8aef26" containerName="revoke-keys" Nov 03 20:03:53.332566 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.332442 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3238700-fb6a-421a-a36f-8dd0ab8aef26" containerName="revoke-keys" Nov 03 20:03:53.336031 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.336009 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.343082 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.342898 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:53.344353 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.344333 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:03:53.357537 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.357511 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q"] Nov 03 20:03:53.517384 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.517351 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2vn\" (UniqueName: \"kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.517563 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.517425 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.517633 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.517607 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.619278 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.619194 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.619457 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.619274 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2vn\" (UniqueName: \"kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.619457 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.619359 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.620024 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.619951 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.621805 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.621779 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.644252 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.644224 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2vn\" (UniqueName: \"kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.648797 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.648780 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:53.950117 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:53.950093 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q"] Nov 03 20:03:53.952024 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:03:53.951998 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0360df_0f5e_4ca4_a3ce_c01a7ad74be9.slice/crio-86548c936ce48e776c0909a4bf9bd8b097b4f5b4f6c2f2b1540044b1669ca572 WatchSource:0}: Error finding container 86548c936ce48e776c0909a4bf9bd8b097b4f5b4f6c2f2b1540044b1669ca572: Status 404 returned error can't find the container with id 86548c936ce48e776c0909a4bf9bd8b097b4f5b4f6c2f2b1540044b1669ca572 Nov 03 20:03:54.257893 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:54.257861 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" event={"ID":"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9","Type":"ContainerStarted","Data":"ec5182379fed0022e06a39cadbebfa310cb15ec8d0b7dd7c46b20f55d079f5f4"} Nov 03 20:03:54.258097 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:54.257913 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" event={"ID":"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9","Type":"ContainerStarted","Data":"86548c936ce48e776c0909a4bf9bd8b097b4f5b4f6c2f2b1540044b1669ca572"} Nov 03 20:03:54.281938 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:54.281888 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" podStartSLOduration=1.281874362 podStartE2EDuration="1.281874362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:54.279519334 +0000 UTC m=+2200.620632833" watchObservedRunningTime="2026-11-03 20:03:54.281874362 +0000 UTC m=+2200.622987847" Nov 03 20:03:55.265851 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:55.265818 5233 generic.go:356] "Generic (PLEG): container finished" podID="ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" containerID="ec5182379fed0022e06a39cadbebfa310cb15ec8d0b7dd7c46b20f55d079f5f4" exitCode=0 Nov 03 20:03:55.266296 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:55.265904 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" event={"ID":"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9","Type":"ContainerDied","Data":"ec5182379fed0022e06a39cadbebfa310cb15ec8d0b7dd7c46b20f55d079f5f4"} Nov 03 20:03:56.408218 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.408188 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:56.447534 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.447504 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca\") pod \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " Nov 03 20:03:56.447707 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.447554 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn\" (UniqueName: \"kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn\") pod \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " Nov 03 20:03:56.447707 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.447688 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca\") pod \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\" (UID: \"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9\") " Nov 03 20:03:56.448295 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.448259 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca" pod "ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" (UID: "ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:56.449887 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.449849 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn" pod "ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" (UID: "ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9"). InnerVolumeSpecName "kube-api-access-qd2vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:56.450032 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.449921 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca" pod "ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" (UID: "ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:56.549142 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.549065 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:56.549142 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.549094 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:56.549142 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:56.549105 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2vn\" (UniqueName: \"kubernetes.io/projected/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9-kube-api-access-qd2vn\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:57.278059 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:57.278028 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" Nov 03 20:03:57.278244 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:57.278076 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q" event={"ID":"ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9","Type":"ContainerDied","Data":"86548c936ce48e776c0909a4bf9bd8b097b4f5b4f6c2f2b1540044b1669ca572"} Nov 03 20:03:57.278244 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:03:57.278131 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86548c936ce48e776c0909a4bf9bd8b097b4f5b4f6c2f2b1540044b1669ca572" Nov 03 20:04:03.451089 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:03.451049 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt"] Nov 03 20:04:03.464107 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:03.464075 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q"] Nov 03 20:04:04.337585 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:04.337550 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q"] Nov 03 20:04:04.340888 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:04.340853 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcdf83-q4h9q"] Nov 03 20:04:05.837427 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:05.837393 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" path="/var/lib/kubelet/pods/ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9/volumes" Nov 03 20:04:08.451525 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.451479 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" podUID="2f46cbf1-0500-4132-acb9-11ed593201ae" containerName="payload-processing" containerID="cri-o://10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16" gracePeriod=25 Nov 03 20:04:08.465067 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.465020 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" podUID="a7d7b260-8d86-4454-b58b-094e167875cc" containerName="payload-pre-processing" containerID="cri-o://26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6" gracePeriod=25 Nov 03 20:04:08.740106 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.740076 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:04:08.761693 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.761668 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:04:08.875619 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.875588 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume\") pod \"a7d7b260-8d86-4454-b58b-094e167875cc\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " Nov 03 20:04:08.875874 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.875627 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg\" (UniqueName: \"kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg\") pod \"2f46cbf1-0500-4132-acb9-11ed593201ae\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " Nov 03 20:04:08.875874 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.875676 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh\" (UniqueName: \"kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh\") pod \"a7d7b260-8d86-4454-b58b-094e167875cc\" (UID: \"a7d7b260-8d86-4454-b58b-094e167875cc\") " Nov 03 20:04:08.875874 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.875785 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume\") pod \"2f46cbf1-0500-4132-acb9-11ed593201ae\" (UID: \"2f46cbf1-0500-4132-acb9-11ed593201ae\") " Nov 03 20:04:08.876124 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.876093 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume" pod "a7d7b260-8d86-4454-b58b-094e167875cc" (UID: "a7d7b260-8d86-4454-b58b-094e167875cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:08.876229 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.876202 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume" pod "2f46cbf1-0500-4132-acb9-11ed593201ae" (UID: "2f46cbf1-0500-4132-acb9-11ed593201ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:08.876498 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.876473 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7b260-8d86-4454-b58b-094e167875cc-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:08.876582 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.876504 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f46cbf1-0500-4132-acb9-11ed593201ae-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:08.877931 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.877909 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh" pod "a7d7b260-8d86-4454-b58b-094e167875cc" (UID: "a7d7b260-8d86-4454-b58b-094e167875cc"). InnerVolumeSpecName "kube-api-access-j7zgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:08.878417 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.878401 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg" pod "2f46cbf1-0500-4132-acb9-11ed593201ae" (UID: "2f46cbf1-0500-4132-acb9-11ed593201ae"). InnerVolumeSpecName "kube-api-access-bfjxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:08.977431 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.977404 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfjxg\" (UniqueName: \"kubernetes.io/projected/2f46cbf1-0500-4132-acb9-11ed593201ae-kube-api-access-bfjxg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:08.977431 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:08.977431 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zgh\" (UniqueName: \"kubernetes.io/projected/a7d7b260-8d86-4454-b58b-094e167875cc-kube-api-access-j7zgh\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:09.360359 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.360273 5233 generic.go:356] "Generic (PLEG): container finished" podID="a7d7b260-8d86-4454-b58b-094e167875cc" containerID="26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6" exitCode=0 Nov 03 20:04:09.360359 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.360319 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" Nov 03 20:04:09.360581 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.360356 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" event={"ID":"a7d7b260-8d86-4454-b58b-094e167875cc","Type":"ContainerDied","Data":"26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6"} Nov 03 20:04:09.360581 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.360410 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q" event={"ID":"a7d7b260-8d86-4454-b58b-094e167875cc","Type":"ContainerDied","Data":"2482eb1a6594a77d31fa2cbfac3129d6602061f9ed3fd2a61cc43275d6f98951"} Nov 03 20:04:09.360581 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.360432 5233 scope.go:122] "RemoveContainer" containerID="26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6" Nov 03 20:04:09.361743 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.361724 5233 generic.go:356] "Generic (PLEG): container finished" podID="2f46cbf1-0500-4132-acb9-11ed593201ae" containerID="10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16" exitCode=0 Nov 03 20:04:09.361843 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.361754 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" event={"ID":"2f46cbf1-0500-4132-acb9-11ed593201ae","Type":"ContainerDied","Data":"10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16"} Nov 03 20:04:09.361843 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.361780 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" Nov 03 20:04:09.361843 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.361803 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt" event={"ID":"2f46cbf1-0500-4132-acb9-11ed593201ae","Type":"ContainerDied","Data":"13b125188055bebc3ca5977107fa8608333740d3d949d45a6adad776e7017984"} Nov 03 20:04:09.371004 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.370988 5233 scope.go:122] "RemoveContainer" containerID="26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6" Nov 03 20:04:09.371248 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:04:09.371229 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6\": container with ID starting with 26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6 not found: ID does not exist" containerID="26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6" Nov 03 20:04:09.371293 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.371257 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6"} err="failed to get container status \"26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6\": rpc error: code = NotFound desc = could not find container \"26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6\": container with ID starting with 26bcad9bcd0c8b99e2befb2a0694b2ff76ce8ad9558a9e992a45ec713a07f2c6 not found: ID does not exist" Nov 03 20:04:09.371293 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.371273 5233 scope.go:122] "RemoveContainer" containerID="10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16" Nov 03 20:04:09.379486 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.379468 5233 scope.go:122] "RemoveContainer" containerID="10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16" Nov 03 20:04:09.379741 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:04:09.379714 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16\": container with ID starting with 10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16 not found: ID does not exist" containerID="10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16" Nov 03 20:04:09.379789 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.379751 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16"} err="failed to get container status \"10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16\": rpc error: code = NotFound desc = could not find container \"10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16\": container with ID starting with 10cbf6baa53d0fe0ac44e7232f8b14b481221e9e2ebeb81fd3243e3356b3eb16 not found: ID does not exist" Nov 03 20:04:09.391210 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.391191 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt"] Nov 03 20:04:09.397299 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.397277 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcdf83-8444d495bt4bnt"] Nov 03 20:04:09.408979 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.408949 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q"] Nov 03 20:04:09.415641 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.415622 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcdf83-56549kw69q"] Nov 03 20:04:09.834816 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.834783 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f46cbf1-0500-4132-acb9-11ed593201ae" path="/var/lib/kubelet/pods/2f46cbf1-0500-4132-acb9-11ed593201ae/volumes" Nov 03 20:04:09.835414 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:09.835392 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d7b260-8d86-4454-b58b-094e167875cc" path="/var/lib/kubelet/pods/a7d7b260-8d86-4454-b58b-094e167875cc/volumes" Nov 03 20:04:21.402399 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:04:21.402356 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:04:49.617651 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.617615 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292"] Nov 03 20:04:49.618162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618056 5233 cpu_manager.go:414] "containerMap: removing container" podUID="2f46cbf1-0500-4132-acb9-11ed593201ae" containerName="payload-processing" Nov 03 20:04:49.618162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618069 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f46cbf1-0500-4132-acb9-11ed593201ae" containerName="payload-processing" Nov 03 20:04:49.618162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618081 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" containerName="revoke-keys" Nov 03 20:04:49.618162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618086 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" containerName="revoke-keys" Nov 03 20:04:49.618162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618094 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a7d7b260-8d86-4454-b58b-094e167875cc" containerName="payload-pre-processing" Nov 03 20:04:49.618162 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618100 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d7b260-8d86-4454-b58b-094e167875cc" containerName="payload-pre-processing" Nov 03 20:04:49.618510 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618176 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0360df-0f5e-4ca4-a3ce-c01a7ad74be9" containerName="revoke-keys" Nov 03 20:04:49.618510 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618187 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d7b260-8d86-4454-b58b-094e167875cc" containerName="payload-pre-processing" Nov 03 20:04:49.618510 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.618199 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f46cbf1-0500-4132-acb9-11ed593201ae" containerName="payload-processing" Nov 03 20:04:49.622996 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.622186 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.629735 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.629707 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.629892 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.629815 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgxp\" (UniqueName: \"kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.630316 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.630276 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.630596 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.630299 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:49.630701 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.630380 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:04:49.634120 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.634097 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292"] Nov 03 20:04:49.731269 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.731231 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.731441 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.731319 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.731441 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.731372 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgxp\" (UniqueName: \"kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.732018 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.731992 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.733702 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.733668 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.744033 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.744003 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgxp\" (UniqueName: \"kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp\") pod \"maas-api-revoke-sub-452b6d7b-m6292\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:49.947355 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:49.947277 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:50.227980 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:50.227947 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292"] Nov 03 20:04:50.230624 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:04:50.230595 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826fbf06_281b_440c_b440_e1f714b28b3b.slice/crio-d1034c7d4fe38327adb1524e4a833f61dff48d1a0ddad0cf0c3ec546b27809c2 WatchSource:0}: Error finding container d1034c7d4fe38327adb1524e4a833f61dff48d1a0ddad0cf0c3ec546b27809c2: Status 404 returned error can't find the container with id d1034c7d4fe38327adb1524e4a833f61dff48d1a0ddad0cf0c3ec546b27809c2 Nov 03 20:04:50.693433 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:50.693333 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" event={"ID":"826fbf06-281b-440c-b440-e1f714b28b3b","Type":"ContainerStarted","Data":"0e1730f72590914d8c07de528eb6c0a9bd43af957b859980beac74490b03b7ef"} Nov 03 20:04:50.693433 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:50.693413 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" event={"ID":"826fbf06-281b-440c-b440-e1f714b28b3b","Type":"ContainerStarted","Data":"d1034c7d4fe38327adb1524e4a833f61dff48d1a0ddad0cf0c3ec546b27809c2"} Nov 03 20:04:50.716181 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:50.715852 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" podStartSLOduration=1.715833887 podStartE2EDuration="1.715833887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:50.711733432 +0000 UTC m=+2257.052846917" watchObservedRunningTime="2026-11-03 20:04:50.715833887 +0000 UTC m=+2257.056947371" Nov 03 20:04:51.290665 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:04:51.290621 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:04:51.705468 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:51.705431 5233 generic.go:356] "Generic (PLEG): container finished" podID="826fbf06-281b-440c-b440-e1f714b28b3b" containerID="0e1730f72590914d8c07de528eb6c0a9bd43af957b859980beac74490b03b7ef" exitCode=0 Nov 03 20:04:51.706056 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:51.705510 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" event={"ID":"826fbf06-281b-440c-b440-e1f714b28b3b","Type":"ContainerDied","Data":"0e1730f72590914d8c07de528eb6c0a9bd43af957b859980beac74490b03b7ef"} Nov 03 20:04:53.811218 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.811184 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:04:53.819417 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.819386 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" event={"ID":"826fbf06-281b-440c-b440-e1f714b28b3b","Type":"ContainerDied","Data":"d1034c7d4fe38327adb1524e4a833f61dff48d1a0ddad0cf0c3ec546b27809c2"} Nov 03 20:04:53.823063 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.823036 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1034c7d4fe38327adb1524e4a833f61dff48d1a0ddad0cf0c3ec546b27809c2" Nov 03 20:04:53.931056 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.925709 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca\") pod \"826fbf06-281b-440c-b440-e1f714b28b3b\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " Nov 03 20:04:53.979341 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.979305 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp\" (UniqueName: \"kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp\") pod \"826fbf06-281b-440c-b440-e1f714b28b3b\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " Nov 03 20:04:53.980169 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.980143 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca\") pod \"826fbf06-281b-440c-b440-e1f714b28b3b\" (UID: \"826fbf06-281b-440c-b440-e1f714b28b3b\") " Nov 03 20:04:53.990535 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:53.990393 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca" pod "826fbf06-281b-440c-b440-e1f714b28b3b" (UID: "826fbf06-281b-440c-b440-e1f714b28b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:54.004507 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:54.004460 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.082565 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:54.082523 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp" pod "826fbf06-281b-440c-b440-e1f714b28b3b" (UID: "826fbf06-281b-440c-b440-e1f714b28b3b"). InnerVolumeSpecName "kube-api-access-kbgxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:54.099275 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:54.099235 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca" pod "826fbf06-281b-440c-b440-e1f714b28b3b" (UID: "826fbf06-281b-440c-b440-e1f714b28b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:54.105734 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:54.105671 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826fbf06-281b-440c-b440-e1f714b28b3b-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.105734 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:54.105707 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbgxp\" (UniqueName: \"kubernetes.io/projected/826fbf06-281b-440c-b440-e1f714b28b3b-kube-api-access-kbgxp\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.825094 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:04:54.825060 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292" Nov 03 20:05:03.390176 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.390132 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts"] Nov 03 20:05:03.390871 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.390848 5233 cpu_manager.go:414] "containerMap: removing container" podUID="826fbf06-281b-440c-b440-e1f714b28b3b" containerName="revoke-keys" Nov 03 20:05:03.391005 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.390873 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826fbf06-281b-440c-b440-e1f714b28b3b" containerName="revoke-keys" Nov 03 20:05:03.391092 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.391007 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="826fbf06-281b-440c-b440-e1f714b28b3b" containerName="revoke-keys" Nov 03 20:05:03.398469 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.398443 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.405005 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.404958 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts"] Nov 03 20:05:03.405294 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.405225 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:03.406331 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.406294 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:05:03.425181 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.425136 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.425456 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.425429 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6cvp\" (UniqueName: \"kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.425610 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.425591 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.546083 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.528568 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.546516 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.546486 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6cvp\" (UniqueName: \"kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.546706 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.546684 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.558994 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.547715 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.569511 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.569480 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.600239 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.600195 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6cvp\" (UniqueName: \"kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp\") pod \"maas-api-revoke-sub-3e31a506-2k2ts\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:03.722366 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:03.722334 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:04.568294 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:05:04.568259 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ee6f15_222b_41d7_92f1_1dba7a12a741.slice/crio-30788e5c151a748efc792bdd6de08c02106840f310101ef5c1388c3e7366578b WatchSource:0}: Error finding container 30788e5c151a748efc792bdd6de08c02106840f310101ef5c1388c3e7366578b: Status 404 returned error can't find the container with id 30788e5c151a748efc792bdd6de08c02106840f310101ef5c1388c3e7366578b Nov 03 20:05:04.593575 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:04.593542 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts"] Nov 03 20:05:04.968332 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:04.968298 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" event={"ID":"27ee6f15-222b-41d7-92f1-1dba7a12a741","Type":"ContainerStarted","Data":"30788e5c151a748efc792bdd6de08c02106840f310101ef5c1388c3e7366578b"} Nov 03 20:05:05.434601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.434067 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:05:05.434601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.434142 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:05.646169 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.646102 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-flqfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:05:05.646792 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.646752 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-flqfr" podUID="a765c74d-cbf2-4d02-8212-a9635bb749ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:05.648174 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.648124 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-fw2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:05:05.648344 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.648323 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-fw2bg" podUID="0693dfcf-c351-424a-99b5-8dd89f1a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:05.982401 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.982357 5233 generic.go:356] "Generic (PLEG): container finished" podID="27ee6f15-222b-41d7-92f1-1dba7a12a741" containerID="0098ee419f9003187fa1cd00db1b55462013c1bf05f4642bee7925d52d1ad798" exitCode=0 Nov 03 20:05:05.982592 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:05.982571 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" event={"ID":"27ee6f15-222b-41d7-92f1-1dba7a12a741","Type":"ContainerDied","Data":"0098ee419f9003187fa1cd00db1b55462013c1bf05f4642bee7925d52d1ad798"} Nov 03 20:05:07.213230 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.213194 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:07.318020 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.317199 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp\" (UniqueName: \"kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp\") pod \"27ee6f15-222b-41d7-92f1-1dba7a12a741\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " Nov 03 20:05:07.318020 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.317298 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca\") pod \"27ee6f15-222b-41d7-92f1-1dba7a12a741\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " Nov 03 20:05:07.318020 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.317404 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca\") pod \"27ee6f15-222b-41d7-92f1-1dba7a12a741\" (UID: \"27ee6f15-222b-41d7-92f1-1dba7a12a741\") " Nov 03 20:05:07.318684 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.318571 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca" pod "27ee6f15-222b-41d7-92f1-1dba7a12a741" (UID: "27ee6f15-222b-41d7-92f1-1dba7a12a741"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:07.321522 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.321478 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca" pod "27ee6f15-222b-41d7-92f1-1dba7a12a741" (UID: "27ee6f15-222b-41d7-92f1-1dba7a12a741"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:07.323137 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.323102 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp" pod "27ee6f15-222b-41d7-92f1-1dba7a12a741" (UID: "27ee6f15-222b-41d7-92f1-1dba7a12a741"). InnerVolumeSpecName "kube-api-access-c6cvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:07.419564 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.419461 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:07.419564 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.419507 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6cvp\" (UniqueName: \"kubernetes.io/projected/27ee6f15-222b-41d7-92f1-1dba7a12a741-kube-api-access-c6cvp\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:07.419564 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:07.419526 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ee6f15-222b-41d7-92f1-1dba7a12a741-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:08.029601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:08.022053 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" event={"ID":"27ee6f15-222b-41d7-92f1-1dba7a12a741","Type":"ContainerDied","Data":"30788e5c151a748efc792bdd6de08c02106840f310101ef5c1388c3e7366578b"} Nov 03 20:05:08.029601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:08.022118 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30788e5c151a748efc792bdd6de08c02106840f310101ef5c1388c3e7366578b" Nov 03 20:05:08.029601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:08.022187 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts" Nov 03 20:05:12.024052 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:12.024016 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn"] Nov 03 20:05:12.027219 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:12.027190 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b306a730-jkjjn"] Nov 03 20:05:13.837648 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:13.837612 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b2131e-8881-4023-b7c2-9b6d9d06ecaa" path="/var/lib/kubelet/pods/c1b2131e-8881-4023-b7c2-9b6d9d06ecaa/volumes" Nov 03 20:05:20.398124 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:20.398082 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-jtkbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:05:20.399666 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:20.399631 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-jtkbc" podUID="dac123bf-79b0-466c-9546-18b94aadcbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:21.276389 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:05:21.276351 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:05:22.860142 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:22.860112 5233 scope.go:122] "RemoveContainer" containerID="69a718b46f41f0a3b6bb6a151eed3356a97bac460b90328fc5af07ce2f5dadbe" Nov 03 20:05:22.871642 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:22.871620 5233 scope.go:122] "RemoveContainer" containerID="6282a598dd1e204d6c7146ae61841544c27e62fea87c8ae5b39914232ced34be" Nov 03 20:05:23.900538 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.900458 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78"] Nov 03 20:05:23.901027 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.900868 5233 cpu_manager.go:414] "containerMap: removing container" podUID="27ee6f15-222b-41d7-92f1-1dba7a12a741" containerName="revoke-keys" Nov 03 20:05:23.901027 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.900879 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ee6f15-222b-41d7-92f1-1dba7a12a741" containerName="revoke-keys" Nov 03 20:05:23.901027 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.900983 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ee6f15-222b-41d7-92f1-1dba7a12a741" containerName="revoke-keys" Nov 03 20:05:23.905786 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.905763 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:23.911905 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.911878 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:23.912176 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.912155 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:05:23.916288 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.915890 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78"] Nov 03 20:05:23.953949 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.953920 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c982g\" (UniqueName: \"kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:23.954093 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.953996 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:23.954135 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:23.954088 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.055363 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.055332 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c982g\" (UniqueName: \"kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.055543 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.055410 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.055543 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.055455 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.056106 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.056083 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.057678 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.057655 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.067474 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.067450 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c982g\" (UniqueName: \"kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g\") pod \"maas-api-revoke-sub-8ef03f88-4bg78\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.225355 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.225332 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:24.504804 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:24.504781 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78"] Nov 03 20:05:24.507220 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:05:24.507192 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a6091e_deb2_4bb3_84b7_5c9b82b39a49.slice/crio-27d98b1e0a6cf804a1dbfb2ad02649593fcc0235aa691186520e63c44e6403a3 WatchSource:0}: Error finding container 27d98b1e0a6cf804a1dbfb2ad02649593fcc0235aa691186520e63c44e6403a3: Status 404 returned error can't find the container with id 27d98b1e0a6cf804a1dbfb2ad02649593fcc0235aa691186520e63c44e6403a3 Nov 03 20:05:25.371551 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:25.371520 5233 generic.go:356] "Generic (PLEG): container finished" podID="f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" containerID="9d6c335d585e50d38ac32b57e53dc95eafc2d466bff423c13ceb2718a6305254" exitCode=0 Nov 03 20:05:25.371897 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:25.371602 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" event={"ID":"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49","Type":"ContainerDied","Data":"9d6c335d585e50d38ac32b57e53dc95eafc2d466bff423c13ceb2718a6305254"} Nov 03 20:05:25.371897 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:25.371657 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" event={"ID":"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49","Type":"ContainerStarted","Data":"27d98b1e0a6cf804a1dbfb2ad02649593fcc0235aa691186520e63c44e6403a3"} Nov 03 20:05:26.524543 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.524517 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:26.579984 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.579938 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca\") pod \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " Nov 03 20:05:26.580130 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.580024 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g\" (UniqueName: \"kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g\") pod \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " Nov 03 20:05:26.580130 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.580067 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca\") pod \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\" (UID: \"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49\") " Nov 03 20:05:26.580314 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.580293 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca" pod "f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" (UID: "f4a6091e-deb2-4bb3-84b7-5c9b82b39a49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:26.580458 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.580436 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:26.582131 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.582105 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca" pod "f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" (UID: "f4a6091e-deb2-4bb3-84b7-5c9b82b39a49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:26.582131 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.582104 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g" pod "f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" (UID: "f4a6091e-deb2-4bb3-84b7-5c9b82b39a49"). InnerVolumeSpecName "kube-api-access-c982g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:26.681832 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.681770 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c982g\" (UniqueName: \"kubernetes.io/projected/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-kube-api-access-c982g\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:26.681832 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:26.681797 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.383722 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:27.383682 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" Nov 03 20:05:27.383722 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:27.383708 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78" event={"ID":"f4a6091e-deb2-4bb3-84b7-5c9b82b39a49","Type":"ContainerDied","Data":"27d98b1e0a6cf804a1dbfb2ad02649593fcc0235aa691186520e63c44e6403a3"} Nov 03 20:05:27.383940 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:27.383757 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d98b1e0a6cf804a1dbfb2ad02649593fcc0235aa691186520e63c44e6403a3" Nov 03 20:05:42.023400 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:42.023361 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g"] Nov 03 20:05:42.026086 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:42.026059 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdr7g"] Nov 03 20:05:43.473692 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.473650 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7"] Nov 03 20:05:43.474333 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.474313 5233 cpu_manager.go:414] "containerMap: removing container" podUID="f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" containerName="revoke-keys" Nov 03 20:05:43.474428 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.474335 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" containerName="revoke-keys" Nov 03 20:05:43.474487 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.474441 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" containerName="revoke-keys" Nov 03 20:05:43.477896 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.477870 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.491275 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.491253 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9eef14db-openshift-default-dockercfg-jftnf\"" Nov 03 20:05:43.493233 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.493211 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7"] Nov 03 20:05:43.543898 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.543859 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544073 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.543908 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544073 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544021 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544161 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544080 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544161 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544128 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544161 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544150 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xf4\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544275 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544245 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544357 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544332 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.544484 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.544404 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.645665 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.645627 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.645873 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.645680 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.645873 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.645699 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xf4\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.645873 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.645807 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.645924 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.645946 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646016 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646047 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646082 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646468 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646326 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646468 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646379 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646468 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646408 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646468 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646459 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.646692 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.646603 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.648068 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.648049 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.648610 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.648587 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.658288 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.658268 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.658512 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.658489 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xf4\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4\") pod \"e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.802079 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.801995 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:43.838397 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:43.838368 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00cd0878-af6f-4f01-a729-5e1da1301d4d" path="/var/lib/kubelet/pods/00cd0878-af6f-4f01-a729-5e1da1301d4d/volumes" Nov 03 20:05:44.094618 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.094590 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7"] Nov 03 20:05:44.096306 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:05:44.096275 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7ebecd_fb38_416f_bc12_705a1e56b1e6.slice/crio-6674eccb592a07c4e92df9b142e801f7ef2fedf9bcfd01d6e6c77c2c9b08e5c0 WatchSource:0}: Error finding container 6674eccb592a07c4e92df9b142e801f7ef2fedf9bcfd01d6e6c77c2c9b08e5c0: Status 404 returned error can't find the container with id 6674eccb592a07c4e92df9b142e801f7ef2fedf9bcfd01d6e6c77c2c9b08e5c0 Nov 03 20:05:44.098597 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.098566 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:44.098676 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.098630 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:44.098676 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.098657 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:44.487286 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.487237 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" event={"ID":"5b7ebecd-fb38-416f-bc12-705a1e56b1e6","Type":"ContainerStarted","Data":"3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83"} Nov 03 20:05:44.487715 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.487309 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" event={"ID":"5b7ebecd-fb38-416f-bc12-705a1e56b1e6","Type":"ContainerStarted","Data":"6674eccb592a07c4e92df9b142e801f7ef2fedf9bcfd01d6e6c77c2c9b08e5c0"} Nov 03 20:05:44.514301 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.514240 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podStartSLOduration=1.514217503 podStartE2EDuration="1.514217503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:44.512588442 +0000 UTC m=+2310.853701930" watchObservedRunningTime="2026-11-03 20:05:44.514217503 +0000 UTC m=+2310.855330990" Nov 03 20:05:44.802796 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:44.802700 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:45.807729 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:45.807698 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:46.500691 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:46.500664 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:46.501539 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:46.501519 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:05:47.368903 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.368865 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz"] Nov 03 20:05:47.379560 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.375046 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.388676 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.388644 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9eef14db\"" Nov 03 20:05:47.389501 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.389477 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9eef14db-dockercfg-5chtk\"" Nov 03 20:05:47.391442 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.391421 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz"] Nov 03 20:05:47.394952 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.394928 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn"] Nov 03 20:05:47.398546 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.398526 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.407018 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.406011 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn"] Nov 03 20:05:47.516995 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.516701 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkln\" (UniqueName: \"kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln\") pod \"payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.516995 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.516785 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume\") pod \"payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.516995 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.516832 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcljg\" (UniqueName: \"kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg\") pod \"payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.516995 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.516861 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.618474 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.618440 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkln\" (UniqueName: \"kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln\") pod \"payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.618668 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.618608 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume\") pod \"payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.618742 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.618666 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcljg\" (UniqueName: \"kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg\") pod \"payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.618742 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.618695 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.619539 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.619491 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.619827 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.619800 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume\") pod \"payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.632910 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.632886 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkln\" (UniqueName: \"kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln\") pod \"payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.633051 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.632997 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcljg\" (UniqueName: \"kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg\") pod \"payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.697394 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.697361 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:47.718050 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.717812 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:47.999640 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:47.995491 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz"] Nov 03 20:05:48.001616 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:05:48.001583 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode732e2e9_2d7f_4b6b_b244_20562f4a9dc3.slice/crio-4b54238656422d3448aa50026c369f4c48ad07aa184f995c8393acccb488b3ae WatchSource:0}: Error finding container 4b54238656422d3448aa50026c369f4c48ad07aa184f995c8393acccb488b3ae: Status 404 returned error can't find the container with id 4b54238656422d3448aa50026c369f4c48ad07aa184f995c8393acccb488b3ae Nov 03 20:05:48.023131 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.023103 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn"] Nov 03 20:05:48.533643 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.533606 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" event={"ID":"72cccd42-e733-4dc1-9871-6d60fc4858c8","Type":"ContainerStarted","Data":"4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20"} Nov 03 20:05:48.534155 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.533672 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" event={"ID":"72cccd42-e733-4dc1-9871-6d60fc4858c8","Type":"ContainerStarted","Data":"d614235c699b2de4afb674ae3a83b78586607945f64973bf7d4121957e530652"} Nov 03 20:05:48.534155 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.533703 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:48.535490 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.535464 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" event={"ID":"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3","Type":"ContainerStarted","Data":"fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d"} Nov 03 20:05:48.535697 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.535509 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" event={"ID":"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3","Type":"ContainerStarted","Data":"4b54238656422d3448aa50026c369f4c48ad07aa184f995c8393acccb488b3ae"} Nov 03 20:05:48.535697 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.535533 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:48.563737 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.563691 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" podStartSLOduration=1.563676281 podStartE2EDuration="1.563676281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:48.561183877 +0000 UTC m=+2314.902297363" watchObservedRunningTime="2026-11-03 20:05:48.563676281 +0000 UTC m=+2314.904789765" Nov 03 20:05:48.588105 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:48.588058 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" podStartSLOduration=1.588046504 podStartE2EDuration="1.588046504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:48.585097936 +0000 UTC m=+2314.926211421" watchObservedRunningTime="2026-11-03 20:05:48.588046504 +0000 UTC m=+2314.929159988" Nov 03 20:05:51.277360 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:05:51.277321 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:05:56.822946 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.822904 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg"] Nov 03 20:05:56.827911 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.827890 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.833942 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.833915 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:56.835173 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.835115 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dx9s4\"" Nov 03 20:05:56.839148 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.839128 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg"] Nov 03 20:05:56.849501 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.849476 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.849607 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.849546 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.849709 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.849683 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcxs\" (UniqueName: \"kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.950413 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.950377 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.950610 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.950445 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcxs\" (UniqueName: \"kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.950671 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.950621 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.951289 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.951264 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.953018 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.952989 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:56.963382 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:56.963354 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcxs\" (UniqueName: \"kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs\") pod \"maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:57.147956 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:57.147862 5233 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" secret="" err="secret \"maas-api-cleanup-dockercfg-dx9s4\" not found" Nov 03 20:05:57.147956 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:57.147930 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:57.436308 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:57.436277 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg"] Nov 03 20:05:57.438206 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:05:57.438168 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce88ce9e_6c4f_4f29_96e6_eeaf733d1a1f.slice/crio-3c6f15f765d5c7515c91e216ab0fd0674c330991cdfcafaac62451ecb347a127 WatchSource:0}: Error finding container 3c6f15f765d5c7515c91e216ab0fd0674c330991cdfcafaac62451ecb347a127: Status 404 returned error can't find the container with id 3c6f15f765d5c7515c91e216ab0fd0674c330991cdfcafaac62451ecb347a127 Nov 03 20:05:57.645274 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:57.645225 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" event={"ID":"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f","Type":"ContainerStarted","Data":"3c6f15f765d5c7515c91e216ab0fd0674c330991cdfcafaac62451ecb347a127"} Nov 03 20:05:58.653793 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:58.653708 5233 generic.go:356] "Generic (PLEG): container finished" podID="ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" containerID="e7c1c2ebf298d9ce56dc16d93ffd0ca974741ae12b888b6a55b7f7b2f8feb6e7" exitCode=0 Nov 03 20:05:58.653793 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:58.653750 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" event={"ID":"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f","Type":"ContainerDied","Data":"e7c1c2ebf298d9ce56dc16d93ffd0ca974741ae12b888b6a55b7f7b2f8feb6e7"} Nov 03 20:05:59.545588 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.545558 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:05:59.547053 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.545622 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:05:59.660707 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.660678 5233 generic.go:356] "Generic (PLEG): container finished" podID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerID="3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83" exitCode=0 Nov 03 20:05:59.661126 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.660754 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" event={"ID":"5b7ebecd-fb38-416f-bc12-705a1e56b1e6","Type":"ContainerDied","Data":"3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83"} Nov 03 20:05:59.661569 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.661241 5233 scope.go:122] "RemoveContainer" containerID="3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83" Nov 03 20:05:59.662453 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.662424 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:59.662577 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.662505 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:59.662577 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.662556 5233 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:59.853282 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.853257 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:05:59.883496 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.883465 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs\" (UniqueName: \"kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs\") pod \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " Nov 03 20:05:59.883684 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.883547 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca\") pod \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " Nov 03 20:05:59.883684 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.883612 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca\") pod \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\" (UID: \"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f\") " Nov 03 20:05:59.884055 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.884023 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca" pod "ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" (UID: "ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:59.886485 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.886461 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs" pod "ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" (UID: "ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f"). InnerVolumeSpecName "kube-api-access-rgcxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:59.886728 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.886572 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca" pod "ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" (UID: "ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:59.985438 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.985400 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgcxs\" (UniqueName: \"kubernetes.io/projected/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-kube-api-access-rgcxs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:59.985438 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.985434 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:59.985438 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:05:59.985443 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:00.668636 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.668601 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" event={"ID":"5b7ebecd-fb38-416f-bc12-705a1e56b1e6","Type":"ContainerStarted","Data":"965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0"} Nov 03 20:06:00.670083 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.670059 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" event={"ID":"ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f","Type":"ContainerDied","Data":"3c6f15f765d5c7515c91e216ab0fd0674c330991cdfcafaac62451ecb347a127"} Nov 03 20:06:00.670083 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.670080 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg" Nov 03 20:06:00.670265 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.670097 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c6f15f765d5c7515c91e216ab0fd0674c330991cdfcafaac62451ecb347a127" Nov 03 20:06:00.805000 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.804665 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:06:00.805000 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.804801 5233 patch_prober.go:29] interesting pod/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:06:00.805000 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:00.804857 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 20:06:01.508996 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:01.508067 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:06:01.806052 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:01.805353 5233 patch_prober.go:29] interesting pod/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:06:01.806052 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:01.805413 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 20:06:02.805995 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:02.805318 5233 patch_prober.go:29] interesting pod/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:06:02.805995 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:02.805378 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 20:06:03.803423 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:03.803390 5233 patch_prober.go:29] interesting pod/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:06:03.803711 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:03.803682 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 20:06:04.803363 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:04.803321 5233 patch_prober.go:29] interesting pod/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:06:04.803991 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:04.803940 5233 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 20:06:05.807335 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:05.807304 5233 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:06:05.808873 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:05.808848 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:06:07.010626 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:07.010595 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz"] Nov 03 20:06:07.025118 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:07.024957 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn"] Nov 03 20:06:07.728833 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:07.728790 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg"] Nov 03 20:06:07.732308 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:07.732284 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eef14db-t7ncg"] Nov 03 20:06:07.836891 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:07.836861 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" path="/var/lib/kubelet/pods/ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f/volumes" Nov 03 20:06:12.010897 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.010849 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" podUID="e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" containerName="payload-processing" containerID="cri-o://fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d" gracePeriod=25 Nov 03 20:06:12.025463 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.025409 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" podUID="72cccd42-e733-4dc1-9871-6d60fc4858c8" containerName="payload-pre-processing" containerID="cri-o://4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20" gracePeriod=25 Nov 03 20:06:12.281678 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.281615 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:06:12.309277 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.309250 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:06:12.338829 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.338796 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln\" (UniqueName: \"kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln\") pod \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " Nov 03 20:06:12.339020 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.338882 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume\") pod \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\" (UID: \"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3\") " Nov 03 20:06:12.339240 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.339218 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume" pod "e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" (UID: "e732e2e9-2d7f-4b6b-b244-20562f4a9dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:12.339334 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.339320 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:12.340858 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.340838 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln" pod "e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" (UID: "e732e2e9-2d7f-4b6b-b244-20562f4a9dc3"). InnerVolumeSpecName "kube-api-access-lhkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:12.440661 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.440630 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg\" (UniqueName: \"kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg\") pod \"72cccd42-e733-4dc1-9871-6d60fc4858c8\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " Nov 03 20:06:12.440810 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.440729 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume\") pod \"72cccd42-e733-4dc1-9871-6d60fc4858c8\" (UID: \"72cccd42-e733-4dc1-9871-6d60fc4858c8\") " Nov 03 20:06:12.440924 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.440912 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkln\" (UniqueName: \"kubernetes.io/projected/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3-kube-api-access-lhkln\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:12.441134 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.441113 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume" pod "72cccd42-e733-4dc1-9871-6d60fc4858c8" (UID: "72cccd42-e733-4dc1-9871-6d60fc4858c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:12.442695 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.442677 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg" pod "72cccd42-e733-4dc1-9871-6d60fc4858c8" (UID: "72cccd42-e733-4dc1-9871-6d60fc4858c8"). InnerVolumeSpecName "kube-api-access-tcljg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:12.542372 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.542301 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cccd42-e733-4dc1-9871-6d60fc4858c8-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:12.542372 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.542327 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcljg\" (UniqueName: \"kubernetes.io/projected/72cccd42-e733-4dc1-9871-6d60fc4858c8-kube-api-access-tcljg\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:12.766936 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.766903 5233 generic.go:356] "Generic (PLEG): container finished" podID="72cccd42-e733-4dc1-9871-6d60fc4858c8" containerID="4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20" exitCode=0 Nov 03 20:06:12.767151 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.766948 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" Nov 03 20:06:12.767151 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.767012 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" event={"ID":"72cccd42-e733-4dc1-9871-6d60fc4858c8","Type":"ContainerDied","Data":"4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20"} Nov 03 20:06:12.767151 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.767064 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn" event={"ID":"72cccd42-e733-4dc1-9871-6d60fc4858c8","Type":"ContainerDied","Data":"d614235c699b2de4afb674ae3a83b78586607945f64973bf7d4121957e530652"} Nov 03 20:06:12.767151 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.767095 5233 scope.go:122] "RemoveContainer" containerID="4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20" Nov 03 20:06:12.768557 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.768532 5233 generic.go:356] "Generic (PLEG): container finished" podID="e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" containerID="fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d" exitCode=0 Nov 03 20:06:12.768696 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.768571 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" event={"ID":"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3","Type":"ContainerDied","Data":"fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d"} Nov 03 20:06:12.768696 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.768625 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" event={"ID":"e732e2e9-2d7f-4b6b-b244-20562f4a9dc3","Type":"ContainerDied","Data":"4b54238656422d3448aa50026c369f4c48ad07aa184f995c8393acccb488b3ae"} Nov 03 20:06:12.768696 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.768581 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz" Nov 03 20:06:12.777939 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.777920 5233 scope.go:122] "RemoveContainer" containerID="4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20" Nov 03 20:06:12.778234 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:06:12.778212 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20\": container with ID starting with 4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20 not found: ID does not exist" containerID="4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20" Nov 03 20:06:12.778319 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.778241 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20"} err="failed to get container status \"4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20\": rpc error: code = NotFound desc = could not find container \"4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20\": container with ID starting with 4a11af511aaf8f2b931a76a009c936c50bf9ac33c63896da2a32f3980713af20 not found: ID does not exist" Nov 03 20:06:12.778319 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.778258 5233 scope.go:122] "RemoveContainer" containerID="fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d" Nov 03 20:06:12.786637 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.786620 5233 scope.go:122] "RemoveContainer" containerID="fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d" Nov 03 20:06:12.786882 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:06:12.786866 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d\": container with ID starting with fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d not found: ID does not exist" containerID="fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d" Nov 03 20:06:12.786943 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.786887 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d"} err="failed to get container status \"fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d\": rpc error: code = NotFound desc = could not find container \"fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d\": container with ID starting with fbadcb700bc21be147324f8e95130b14a72c24421e3e9a149e9c6ac0a8ba093d not found: ID does not exist" Nov 03 20:06:12.796060 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.796000 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz"] Nov 03 20:06:12.799455 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.799432 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eef14db-6f5c549c55-kvrvz"] Nov 03 20:06:12.811767 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.811735 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn"] Nov 03 20:06:12.815556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:12.815533 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eef14db-7f7bfdbc75-zgxqn"] Nov 03 20:06:13.216681 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:13.216650 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7"] Nov 03 20:06:13.217140 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:13.216863 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" containerID="cri-o://965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0" gracePeriod=30 Nov 03 20:06:13.840711 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:13.840682 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cccd42-e733-4dc1-9871-6d60fc4858c8" path="/var/lib/kubelet/pods/72cccd42-e733-4dc1-9871-6d60fc4858c8/volumes" Nov 03 20:06:13.841145 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:13.841131 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" path="/var/lib/kubelet/pods/e732e2e9-2d7f-4b6b-b244-20562f4a9dc3/volumes" Nov 03 20:06:18.469553 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.469528 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:06:18.606842 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.606769 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.606842 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.606815 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.606842 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.606838 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.606858 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.606901 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.606966 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607031 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607073 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:18.607183 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607141 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607522 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607223 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo\") pod \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\" (UID: \"5b7ebecd-fb38-416f-bc12-705a1e56b1e6\") " Nov 03 20:06:18.607522 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607311 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:18.607522 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607320 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:18.607522 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607361 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:18.607691 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607532 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:18.607691 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607675 5233 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-credential-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.607801 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607696 5233 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istiod-ca-cert\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.607801 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607711 5233 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-socket\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.607801 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607728 5233 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-data\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.607801 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.607742 5233 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-workload-certs\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.609226 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.609195 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:18.609226 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.609196 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:18.609372 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.609294 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "kube-api-access-84xf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:18.609372 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.609354 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo" pod "5b7ebecd-fb38-416f-bc12-705a1e56b1e6" (UID: "5b7ebecd-fb38-416f-bc12-705a1e56b1e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:06:18.708391 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.708361 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xf4\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-kube-api-access-84xf4\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.708391 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.708390 5233 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-envoy\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.708538 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.708399 5233 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-token\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.708538 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.708409 5233 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7ebecd-fb38-416f-bc12-705a1e56b1e6-istio-podinfo\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:18.807923 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.807894 5233 generic.go:356] "Generic (PLEG): container finished" podID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerID="965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0" exitCode=0 Nov 03 20:06:18.808088 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.807940 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" event={"ID":"5b7ebecd-fb38-416f-bc12-705a1e56b1e6","Type":"ContainerDied","Data":"965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0"} Nov 03 20:06:18.808088 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.807944 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" Nov 03 20:06:18.808088 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.808008 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7" event={"ID":"5b7ebecd-fb38-416f-bc12-705a1e56b1e6","Type":"ContainerDied","Data":"6674eccb592a07c4e92df9b142e801f7ef2fedf9bcfd01d6e6c77c2c9b08e5c0"} Nov 03 20:06:18.808088 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.808031 5233 scope.go:122] "RemoveContainer" containerID="965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0" Nov 03 20:06:18.818110 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.818086 5233 scope.go:122] "RemoveContainer" containerID="3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83" Nov 03 20:06:18.826634 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.826614 5233 scope.go:122] "RemoveContainer" containerID="965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0" Nov 03 20:06:18.826914 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:06:18.826888 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0\": container with ID starting with 965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0 not found: ID does not exist" containerID="965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0" Nov 03 20:06:18.826964 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.826923 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0"} err="failed to get container status \"965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0\": rpc error: code = NotFound desc = could not find container \"965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0\": container with ID starting with 965bc17a3073879ef1ced0e97042d39c49b59ccafe17c6ce9a765aabc44534c0 not found: ID does not exist" Nov 03 20:06:18.826964 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.826940 5233 scope.go:122] "RemoveContainer" containerID="3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83" Nov 03 20:06:18.827252 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:06:18.827237 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83\": container with ID starting with 3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83 not found: ID does not exist" containerID="3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83" Nov 03 20:06:18.827298 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.827255 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83"} err="failed to get container status \"3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83\": rpc error: code = NotFound desc = could not find container \"3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83\": container with ID starting with 3b5ae6a5be75f19d9996178267fb78250736a650fad88a7fb0e789d02dab7a83 not found: ID does not exist" Nov 03 20:06:18.832683 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.832661 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7"] Nov 03 20:06:18.837341 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:18.837322 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9eef14db-openshift-default-7667b4556f-tsdv7"] Nov 03 20:06:19.130130 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130095 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs"] Nov 03 20:06:19.130533 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130520 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" Nov 03 20:06:19.130583 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130534 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" Nov 03 20:06:19.130583 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130546 5233 cpu_manager.go:414] "containerMap: removing container" podUID="72cccd42-e733-4dc1-9871-6d60fc4858c8" containerName="payload-pre-processing" Nov 03 20:06:19.130583 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130551 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cccd42-e733-4dc1-9871-6d60fc4858c8" containerName="payload-pre-processing" Nov 03 20:06:19.130583 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130569 5233 cpu_manager.go:414] "containerMap: removing container" podUID="ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" containerName="revoke-keys" Nov 03 20:06:19.130583 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130573 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" containerName="revoke-keys" Nov 03 20:06:19.130583 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130582 5233 cpu_manager.go:414] "containerMap: removing container" podUID="e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" containerName="payload-processing" Nov 03 20:06:19.130763 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130587 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" containerName="payload-processing" Nov 03 20:06:19.130763 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130662 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" Nov 03 20:06:19.130763 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130670 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cccd42-e733-4dc1-9871-6d60fc4858c8" containerName="payload-pre-processing" Nov 03 20:06:19.130763 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130680 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce88ce9e-6c4f-4f29-96e6-eeaf733d1a1f" containerName="revoke-keys" Nov 03 20:06:19.130763 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.130685 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="e732e2e9-2d7f-4b6b-b244-20562f4a9dc3" containerName="payload-processing" Nov 03 20:06:19.135656 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.135636 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.143377 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.143226 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-66e889\"" Nov 03 20:06:19.143377 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.143346 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs"] Nov 03 20:06:19.179315 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.179286 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-66e889-dockercfg-sjgb9\"" Nov 03 20:06:19.187834 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.187806 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l"] Nov 03 20:06:19.188353 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.188336 5233 cpu_manager.go:414] "containerMap: removing container" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" Nov 03 20:06:19.188401 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.188354 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" Nov 03 20:06:19.188445 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.188424 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" containerName="istio-proxy" Nov 03 20:06:19.191571 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.191554 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.202471 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.202447 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l"] Nov 03 20:06:19.214016 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.213950 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.214139 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.214055 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kx7z\" (UniqueName: \"kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z\") pod \"payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.314924 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.314880 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvwm\" (UniqueName: \"kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm\") pod \"payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.315127 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.315057 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.315127 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.315097 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx7z\" (UniqueName: \"kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z\") pod \"payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.315272 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.315138 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.315817 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.315789 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.327811 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.327780 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx7z\" (UniqueName: \"kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z\") pod \"payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.416608 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.416537 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.416768 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.416613 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvwm\" (UniqueName: \"kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm\") pod \"payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.417226 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.417204 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.428800 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.428778 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvwm\" (UniqueName: \"kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm\") pod \"payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.454318 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.454295 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:19.505811 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.505778 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:19.741165 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.741133 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs"] Nov 03 20:06:19.742535 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:06:19.742508 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23dd467_dfbc_428f_bb7e_5101ffb61a09.slice/crio-0993edfc0811f58a869e092e15cb14bbcdcacaea017414799aa8affdf03bd260 WatchSource:0}: Error finding container 0993edfc0811f58a869e092e15cb14bbcdcacaea017414799aa8affdf03bd260: Status 404 returned error can't find the container with id 0993edfc0811f58a869e092e15cb14bbcdcacaea017414799aa8affdf03bd260 Nov 03 20:06:19.808582 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.808311 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l"] Nov 03 20:06:19.813085 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:06:19.813052 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153638a1_0716_48be_b689_9d70976d4cd0.slice/crio-d4dd00ddda20b846b648828d01af5a63901af0c33e5dfbb38db7b3e22fcf91c2 WatchSource:0}: Error finding container d4dd00ddda20b846b648828d01af5a63901af0c33e5dfbb38db7b3e22fcf91c2: Status 404 returned error can't find the container with id d4dd00ddda20b846b648828d01af5a63901af0c33e5dfbb38db7b3e22fcf91c2 Nov 03 20:06:19.816308 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.816232 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" event={"ID":"a23dd467-dfbc-428f-bb7e-5101ffb61a09","Type":"ContainerStarted","Data":"0993edfc0811f58a869e092e15cb14bbcdcacaea017414799aa8affdf03bd260"} Nov 03 20:06:19.845401 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:19.845369 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7ebecd-fb38-416f-bc12-705a1e56b1e6" path="/var/lib/kubelet/pods/5b7ebecd-fb38-416f-bc12-705a1e56b1e6/volumes" Nov 03 20:06:20.826682 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.826640 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" event={"ID":"153638a1-0716-48be-b689-9d70976d4cd0","Type":"ContainerStarted","Data":"2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6"} Nov 03 20:06:20.827191 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.826710 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" event={"ID":"153638a1-0716-48be-b689-9d70976d4cd0","Type":"ContainerStarted","Data":"d4dd00ddda20b846b648828d01af5a63901af0c33e5dfbb38db7b3e22fcf91c2"} Nov 03 20:06:20.827191 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.826738 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:20.828475 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.828438 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" event={"ID":"a23dd467-dfbc-428f-bb7e-5101ffb61a09","Type":"ContainerStarted","Data":"eaf15bf09b738d1397f839a86008ccc0226bb810fc434239e710e209ba52da02"} Nov 03 20:06:20.828613 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.828507 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:20.849486 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.849437 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" podStartSLOduration=1.849420631 podStartE2EDuration="1.849420631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:20.846106098 +0000 UTC m=+2347.187219584" watchObservedRunningTime="2026-11-03 20:06:20.849420631 +0000 UTC m=+2347.190534116" Nov 03 20:06:20.867188 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:20.867140 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" podStartSLOduration=1.867128334 podStartE2EDuration="1.867128334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:20.864324861 +0000 UTC m=+2347.205438348" watchObservedRunningTime="2026-11-03 20:06:20.867128334 +0000 UTC m=+2347.208241839" Nov 03 20:06:21.279294 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:06:21.279259 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:06:22.900422 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:22.900394 5233 scope.go:122] "RemoveContainer" containerID="4a32091e416beaada586356f3ae95aee90c6c058ce58e4e77254142f844204fa" Nov 03 20:06:22.915392 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:22.915365 5233 scope.go:122] "RemoveContainer" containerID="e2dfeba3c59cc827ae3a4016913a5e03524b155d7f1b83e5c14d3fb93343f93f" Nov 03 20:06:31.840445 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:31.840415 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:06:31.841036 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:06:31.840482 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:06:51.277118 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:06:51.277081 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:07:21.277526 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:07:21.277490 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:07:22.969241 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:22.969211 5233 scope.go:122] "RemoveContainer" containerID="c3c83de0ea53e77dc10ce2d356a376a6a04e60b9e583b1cf5fedadf91f83fcb2" Nov 03 20:07:23.026285 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:23.026258 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4"] Nov 03 20:07:23.034985 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:23.031995 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qbkt4"] Nov 03 20:07:23.839071 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:23.838933 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244893ed-5770-407b-8491-007710541082" path="/var/lib/kubelet/pods/244893ed-5770-407b-8491-007710541082/volumes" Nov 03 20:07:42.023260 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:42.023220 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69"] Nov 03 20:07:42.026018 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:42.025995 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jvl69"] Nov 03 20:07:43.835843 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:43.835806 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7adeb9-326a-4193-8bac-741fe3b6ec0c" path="/var/lib/kubelet/pods/9c7adeb9-326a-4193-8bac-741fe3b6ec0c/volumes" Nov 03 20:07:51.276873 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:07:51.276836 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:07:53.076392 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.076353 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn"] Nov 03 20:07:53.080317 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.080298 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.087085 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.087045 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:07:53.087958 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.087516 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ktm9v\"" Nov 03 20:07:53.091950 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.091926 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn"] Nov 03 20:07:53.188458 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.188430 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.188619 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.188496 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.188619 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.188585 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.289401 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.289359 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.289559 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.289414 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.289559 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.289479 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.290036 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.290014 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.291804 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.291774 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.301601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.301578 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d\") pod \"maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.396605 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.396519 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:53.680623 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:53.680594 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn"] Nov 03 20:07:53.682982 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:07:53.682938 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038f1ec1_1d52_45a2_b506_0009988ac557.slice/crio-abf225b673aa274109c1d4d76152782e2b7bc9b506880d6134f109fd8e5f79c8 WatchSource:0}: Error finding container abf225b673aa274109c1d4d76152782e2b7bc9b506880d6134f109fd8e5f79c8: Status 404 returned error can't find the container with id abf225b673aa274109c1d4d76152782e2b7bc9b506880d6134f109fd8e5f79c8 Nov 03 20:07:54.377226 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:54.377133 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" event={"ID":"038f1ec1-1d52-45a2-b506-0009988ac557","Type":"ContainerStarted","Data":"3efebbd59349e2ce62d9232f0d6424f200140176b7bb2e8566ac22d93cc26e7b"} Nov 03 20:07:54.377226 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:54.377199 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" event={"ID":"038f1ec1-1d52-45a2-b506-0009988ac557","Type":"ContainerStarted","Data":"abf225b673aa274109c1d4d76152782e2b7bc9b506880d6134f109fd8e5f79c8"} Nov 03 20:07:54.396352 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:54.396280 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" podStartSLOduration=1.39626455 podStartE2EDuration="1.39626455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:54.395820025 +0000 UTC m=+2440.736933510" watchObservedRunningTime="2026-11-03 20:07:54.39626455 +0000 UTC m=+2440.737378034" Nov 03 20:07:55.384479 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:55.384447 5233 generic.go:356] "Generic (PLEG): container finished" podID="038f1ec1-1d52-45a2-b506-0009988ac557" containerID="3efebbd59349e2ce62d9232f0d6424f200140176b7bb2e8566ac22d93cc26e7b" exitCode=0 Nov 03 20:07:55.384878 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:55.384483 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" event={"ID":"038f1ec1-1d52-45a2-b506-0009988ac557","Type":"ContainerDied","Data":"3efebbd59349e2ce62d9232f0d6424f200140176b7bb2e8566ac22d93cc26e7b"} Nov 03 20:07:56.525075 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.525051 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:56.620784 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.620754 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca\") pod \"038f1ec1-1d52-45a2-b506-0009988ac557\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " Nov 03 20:07:56.620959 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.620797 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d\") pod \"038f1ec1-1d52-45a2-b506-0009988ac557\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " Nov 03 20:07:56.620959 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.620849 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca\") pod \"038f1ec1-1d52-45a2-b506-0009988ac557\" (UID: \"038f1ec1-1d52-45a2-b506-0009988ac557\") " Nov 03 20:07:56.621302 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.621277 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca" pod "038f1ec1-1d52-45a2-b506-0009988ac557" (UID: "038f1ec1-1d52-45a2-b506-0009988ac557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:56.623010 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.622981 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca" pod "038f1ec1-1d52-45a2-b506-0009988ac557" (UID: "038f1ec1-1d52-45a2-b506-0009988ac557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:56.623010 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.622993 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d" pod "038f1ec1-1d52-45a2-b506-0009988ac557" (UID: "038f1ec1-1d52-45a2-b506-0009988ac557"). InnerVolumeSpecName "kube-api-access-tlx8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:56.721899 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.721876 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:56.721899 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.721898 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/038f1ec1-1d52-45a2-b506-0009988ac557-kube-api-access-tlx8d\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:56.722093 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:56.721910 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038f1ec1-1d52-45a2-b506-0009988ac557-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:57.397390 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:57.397362 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" Nov 03 20:07:57.397569 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:57.397386 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn" event={"ID":"038f1ec1-1d52-45a2-b506-0009988ac557","Type":"ContainerDied","Data":"abf225b673aa274109c1d4d76152782e2b7bc9b506880d6134f109fd8e5f79c8"} Nov 03 20:07:57.397569 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:07:57.397452 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf225b673aa274109c1d4d76152782e2b7bc9b506880d6134f109fd8e5f79c8" Nov 03 20:08:03.202850 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:03.202807 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l"] Nov 03 20:08:03.215825 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:03.215779 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs"] Nov 03 20:08:08.204128 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.204069 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" podUID="153638a1-0716-48be-b689-9d70976d4cd0" containerName="payload-processing" containerID="cri-o://2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6" gracePeriod=25 Nov 03 20:08:08.216720 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.216687 5233 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" podUID="a23dd467-dfbc-428f-bb7e-5101ffb61a09" containerName="payload-pre-processing" containerID="cri-o://eaf15bf09b738d1397f839a86008ccc0226bb810fc434239e710e209ba52da02" gracePeriod=25 Nov 03 20:08:08.461512 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.461449 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:08:08.462213 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.462190 5233 generic.go:356] "Generic (PLEG): container finished" podID="153638a1-0716-48be-b689-9d70976d4cd0" containerID="2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6" exitCode=0 Nov 03 20:08:08.462325 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.462265 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" event={"ID":"153638a1-0716-48be-b689-9d70976d4cd0","Type":"ContainerDied","Data":"2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6"} Nov 03 20:08:08.462371 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.462331 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" event={"ID":"153638a1-0716-48be-b689-9d70976d4cd0","Type":"ContainerDied","Data":"d4dd00ddda20b846b648828d01af5a63901af0c33e5dfbb38db7b3e22fcf91c2"} Nov 03 20:08:08.462371 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.462359 5233 scope.go:122] "RemoveContainer" containerID="2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6" Nov 03 20:08:08.464359 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.464338 5233 generic.go:356] "Generic (PLEG): container finished" podID="a23dd467-dfbc-428f-bb7e-5101ffb61a09" containerID="eaf15bf09b738d1397f839a86008ccc0226bb810fc434239e710e209ba52da02" exitCode=0 Nov 03 20:08:08.464459 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.464371 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" event={"ID":"a23dd467-dfbc-428f-bb7e-5101ffb61a09","Type":"ContainerDied","Data":"eaf15bf09b738d1397f839a86008ccc0226bb810fc434239e710e209ba52da02"} Nov 03 20:08:08.471959 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.471943 5233 scope.go:122] "RemoveContainer" containerID="2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6" Nov 03 20:08:08.472276 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:08:08.472252 5233 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6\": container with ID starting with 2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6 not found: ID does not exist" containerID="2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6" Nov 03 20:08:08.472354 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.472286 5233 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6"} err="failed to get container status \"2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6\": rpc error: code = NotFound desc = could not find container \"2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6\": container with ID starting with 2f515e7030986623766976fec4e7ff08e674972dfa16a1c207718b175e3594e6 not found: ID does not exist" Nov 03 20:08:08.513651 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.513621 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:08:08.541409 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.541376 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm\" (UniqueName: \"kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm\") pod \"153638a1-0716-48be-b689-9d70976d4cd0\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " Nov 03 20:08:08.541578 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.541521 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume\") pod \"153638a1-0716-48be-b689-9d70976d4cd0\" (UID: \"153638a1-0716-48be-b689-9d70976d4cd0\") " Nov 03 20:08:08.542143 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.542113 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume" pod "153638a1-0716-48be-b689-9d70976d4cd0" (UID: "153638a1-0716-48be-b689-9d70976d4cd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:08.543752 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.543725 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm" pod "153638a1-0716-48be-b689-9d70976d4cd0" (UID: "153638a1-0716-48be-b689-9d70976d4cd0"). InnerVolumeSpecName "kube-api-access-qhvwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:08.642211 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.642175 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume\") pod \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " Nov 03 20:08:08.642356 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.642230 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z\" (UniqueName: \"kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z\") pod \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\" (UID: \"a23dd467-dfbc-428f-bb7e-5101ffb61a09\") " Nov 03 20:08:08.642548 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.642509 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume" pod "a23dd467-dfbc-428f-bb7e-5101ffb61a09" (UID: "a23dd467-dfbc-428f-bb7e-5101ffb61a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:08.642548 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.642528 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhvwm\" (UniqueName: \"kubernetes.io/projected/153638a1-0716-48be-b689-9d70976d4cd0-kube-api-access-qhvwm\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:08.642548 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.642542 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153638a1-0716-48be-b689-9d70976d4cd0-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:08.644255 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.644235 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z" pod "a23dd467-dfbc-428f-bb7e-5101ffb61a09" (UID: "a23dd467-dfbc-428f-bb7e-5101ffb61a09"). InnerVolumeSpecName "kube-api-access-2kx7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:08.743458 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.743426 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kx7z\" (UniqueName: \"kubernetes.io/projected/a23dd467-dfbc-428f-bb7e-5101ffb61a09-kube-api-access-2kx7z\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:08.743458 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:08.743452 5233 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23dd467-dfbc-428f-bb7e-5101ffb61a09-plugins-config-volume\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.185484 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.185406 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn"] Nov 03 20:08:09.187705 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.187681 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-66e889-spggn"] Nov 03 20:08:09.471560 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.471523 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" Nov 03 20:08:09.472030 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.471522 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs" event={"ID":"a23dd467-dfbc-428f-bb7e-5101ffb61a09","Type":"ContainerDied","Data":"0993edfc0811f58a869e092e15cb14bbcdcacaea017414799aa8affdf03bd260"} Nov 03 20:08:09.472030 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.471705 5233 scope.go:122] "RemoveContainer" containerID="eaf15bf09b738d1397f839a86008ccc0226bb810fc434239e710e209ba52da02" Nov 03 20:08:09.473122 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.472780 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l" Nov 03 20:08:09.542478 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.542445 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs"] Nov 03 20:08:09.580483 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.580456 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-66e889-768ff79b4c-bvtxs"] Nov 03 20:08:09.630047 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.629954 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l"] Nov 03 20:08:09.639659 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.639635 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-66e889-6bbfcc7859-88z9l"] Nov 03 20:08:09.835326 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.835248 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038f1ec1-1d52-45a2-b506-0009988ac557" path="/var/lib/kubelet/pods/038f1ec1-1d52-45a2-b506-0009988ac557/volumes" Nov 03 20:08:09.835628 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.835615 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153638a1-0716-48be-b689-9d70976d4cd0" path="/var/lib/kubelet/pods/153638a1-0716-48be-b689-9d70976d4cd0/volumes" Nov 03 20:08:09.835951 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:09.835939 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23dd467-dfbc-428f-bb7e-5101ffb61a09" path="/var/lib/kubelet/pods/a23dd467-dfbc-428f-bb7e-5101ffb61a09/volumes" Nov 03 20:08:21.112136 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:21.112079 5233 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-8tfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:21.112821 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:21.112770 5233 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-8tfdw" podUID="d6929ca2-35e3-40c0-8010-49b9dc0a68c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:21.420365 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:08:21.420262 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:08:22.992542 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:22.992509 5233 scope.go:122] "RemoveContainer" containerID="a4c7731681f05159d4c601c1aaef58d75dc1c939cc06b2574659b51112a09212" Nov 03 20:08:31.426008 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.425956 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5"] Nov 03 20:08:31.426562 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426543 5233 cpu_manager.go:414] "containerMap: removing container" podUID="153638a1-0716-48be-b689-9d70976d4cd0" containerName="payload-processing" Nov 03 20:08:31.426620 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426564 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153638a1-0716-48be-b689-9d70976d4cd0" containerName="payload-processing" Nov 03 20:08:31.426620 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426580 5233 cpu_manager.go:414] "containerMap: removing container" podUID="a23dd467-dfbc-428f-bb7e-5101ffb61a09" containerName="payload-pre-processing" Nov 03 20:08:31.426620 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426585 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23dd467-dfbc-428f-bb7e-5101ffb61a09" containerName="payload-pre-processing" Nov 03 20:08:31.426620 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426597 5233 cpu_manager.go:414] "containerMap: removing container" podUID="038f1ec1-1d52-45a2-b506-0009988ac557" containerName="revoke-keys" Nov 03 20:08:31.426620 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426602 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038f1ec1-1d52-45a2-b506-0009988ac557" containerName="revoke-keys" Nov 03 20:08:31.426787 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426705 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="153638a1-0716-48be-b689-9d70976d4cd0" containerName="payload-processing" Nov 03 20:08:31.426787 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426716 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23dd467-dfbc-428f-bb7e-5101ffb61a09" containerName="payload-pre-processing" Nov 03 20:08:31.426787 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.426721 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="038f1ec1-1d52-45a2-b506-0009988ac557" containerName="revoke-keys" Nov 03 20:08:31.431633 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.431615 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.436309 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.436283 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:31.436450 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.436334 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ktm9v\"" Nov 03 20:08:31.438515 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.438491 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5"] Nov 03 20:08:31.464345 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.464318 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.464464 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.464379 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.464522 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.464503 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr72n\" (UniqueName: \"kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.565236 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.565198 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr72n\" (UniqueName: \"kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.565475 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.565316 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.565475 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.565348 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.566055 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.566032 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.567706 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.567683 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.577915 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.577895 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr72n\" (UniqueName: \"kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n\") pod \"maas-api-revoke-sub-f03629db-dhhg5\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:31.747707 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:31.747673 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:32.025826 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:32.025801 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5"] Nov 03 20:08:32.028215 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:08:32.028176 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efceb7c_4394_442b_bc5b_72e0cba5d92c.slice/crio-16c22798fe6b4150125dd00a4afa7149fd350de1103fc3dacc3a938b473f5be4 WatchSource:0}: Error finding container 16c22798fe6b4150125dd00a4afa7149fd350de1103fc3dacc3a938b473f5be4: Status 404 returned error can't find the container with id 16c22798fe6b4150125dd00a4afa7149fd350de1103fc3dacc3a938b473f5be4 Nov 03 20:08:32.694668 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:32.694591 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" event={"ID":"0efceb7c-4394-442b-bc5b-72e0cba5d92c","Type":"ContainerStarted","Data":"e6a8782b7bbddf94143e71578c392302b25e4bd3d8e2e682cf08aee7d986bf8b"} Nov 03 20:08:32.694668 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:32.694642 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" event={"ID":"0efceb7c-4394-442b-bc5b-72e0cba5d92c","Type":"ContainerStarted","Data":"16c22798fe6b4150125dd00a4afa7149fd350de1103fc3dacc3a938b473f5be4"} Nov 03 20:08:32.715409 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:32.715345 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" podStartSLOduration=1.7153265869999998 podStartE2EDuration="1.715326587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:32.712586954 +0000 UTC m=+2479.053700441" watchObservedRunningTime="2026-11-03 20:08:32.715326587 +0000 UTC m=+2479.056440074" Nov 03 20:08:33.701528 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:33.701493 5233 generic.go:356] "Generic (PLEG): container finished" podID="0efceb7c-4394-442b-bc5b-72e0cba5d92c" containerID="e6a8782b7bbddf94143e71578c392302b25e4bd3d8e2e682cf08aee7d986bf8b" exitCode=0 Nov 03 20:08:33.702093 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:33.701566 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" event={"ID":"0efceb7c-4394-442b-bc5b-72e0cba5d92c","Type":"ContainerDied","Data":"e6a8782b7bbddf94143e71578c392302b25e4bd3d8e2e682cf08aee7d986bf8b"} Nov 03 20:08:34.844806 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.844781 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:34.902818 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.902787 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n\" (UniqueName: \"kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n\") pod \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " Nov 03 20:08:34.903010 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.902851 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca\") pod \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " Nov 03 20:08:34.903095 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.903032 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca\") pod \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\" (UID: \"0efceb7c-4394-442b-bc5b-72e0cba5d92c\") " Nov 03 20:08:34.903407 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.903375 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca" pod "0efceb7c-4394-442b-bc5b-72e0cba5d92c" (UID: "0efceb7c-4394-442b-bc5b-72e0cba5d92c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:34.904946 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.904917 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca" pod "0efceb7c-4394-442b-bc5b-72e0cba5d92c" (UID: "0efceb7c-4394-442b-bc5b-72e0cba5d92c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:34.905084 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:34.905013 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n" pod "0efceb7c-4394-442b-bc5b-72e0cba5d92c" (UID: "0efceb7c-4394-442b-bc5b-72e0cba5d92c"). InnerVolumeSpecName "kube-api-access-jr72n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:35.004856 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:35.004829 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr72n\" (UniqueName: \"kubernetes.io/projected/0efceb7c-4394-442b-bc5b-72e0cba5d92c-kube-api-access-jr72n\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:35.004856 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:35.004854 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:35.005085 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:35.004862 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0efceb7c-4394-442b-bc5b-72e0cba5d92c-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:35.714260 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:35.714222 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" Nov 03 20:08:35.714260 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:35.714250 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dhhg5" event={"ID":"0efceb7c-4394-442b-bc5b-72e0cba5d92c","Type":"ContainerDied","Data":"16c22798fe6b4150125dd00a4afa7149fd350de1103fc3dacc3a938b473f5be4"} Nov 03 20:08:35.714485 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:35.714328 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c22798fe6b4150125dd00a4afa7149fd350de1103fc3dacc3a938b473f5be4" Nov 03 20:08:46.056051 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:46.056017 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8"] Nov 03 20:08:46.081161 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:46.081129 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712657a1-bhbc8"] Nov 03 20:08:47.841965 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.841930 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3238700-fb6a-421a-a36f-8dd0ab8aef26" path="/var/lib/kubelet/pods/d3238700-fb6a-421a-a36f-8dd0ab8aef26/volumes" Nov 03 20:08:47.879465 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.879433 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj"] Nov 03 20:08:47.879837 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.879825 5233 cpu_manager.go:414] "containerMap: removing container" podUID="0efceb7c-4394-442b-bc5b-72e0cba5d92c" containerName="revoke-keys" Nov 03 20:08:47.879882 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.879838 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efceb7c-4394-442b-bc5b-72e0cba5d92c" containerName="revoke-keys" Nov 03 20:08:47.879935 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.879927 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efceb7c-4394-442b-bc5b-72e0cba5d92c" containerName="revoke-keys" Nov 03 20:08:47.883236 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.883214 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:47.887603 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.887323 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:47.888743 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.888721 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ktm9v\"" Nov 03 20:08:47.892525 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.892499 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj"] Nov 03 20:08:47.984469 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.984437 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dc8d\" (UniqueName: \"kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:47.984469 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.984472 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:47.984699 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:47.984647 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.085559 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.085527 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dc8d\" (UniqueName: \"kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.085698 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.085562 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.085698 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.085656 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.086380 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.086347 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.088010 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.087990 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.098032 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.097961 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dc8d\" (UniqueName: \"kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d\") pod \"maas-api-revoke-sub-0165d7d2-4mbdj\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.199732 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.199707 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:48.476776 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.476750 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj"] Nov 03 20:08:48.479385 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:08:48.479352 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6955eed5_7e8a_472f_a007_df4979408183.slice/crio-922d77af655736f145b0a54f55c3e162a64247c45afb7bc52986b623390bc773 WatchSource:0}: Error finding container 922d77af655736f145b0a54f55c3e162a64247c45afb7bc52986b623390bc773: Status 404 returned error can't find the container with id 922d77af655736f145b0a54f55c3e162a64247c45afb7bc52986b623390bc773 Nov 03 20:08:48.815283 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.815254 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" event={"ID":"6955eed5-7e8a-472f-a007-df4979408183","Type":"ContainerStarted","Data":"247c93102e37cb838fe7535ef0e685d264daf634816747cbec582a6eea2748ee"} Nov 03 20:08:48.815402 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.815304 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" event={"ID":"6955eed5-7e8a-472f-a007-df4979408183","Type":"ContainerStarted","Data":"922d77af655736f145b0a54f55c3e162a64247c45afb7bc52986b623390bc773"} Nov 03 20:08:48.835922 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:48.835881 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" podStartSLOduration=1.835867632 podStartE2EDuration="1.835867632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:48.833093039 +0000 UTC m=+2495.174206521" watchObservedRunningTime="2026-11-03 20:08:48.835867632 +0000 UTC m=+2495.176981117" Nov 03 20:08:49.822333 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:49.822302 5233 generic.go:356] "Generic (PLEG): container finished" podID="6955eed5-7e8a-472f-a007-df4979408183" containerID="247c93102e37cb838fe7535ef0e685d264daf634816747cbec582a6eea2748ee" exitCode=0 Nov 03 20:08:49.822736 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:49.822382 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" event={"ID":"6955eed5-7e8a-472f-a007-df4979408183","Type":"ContainerDied","Data":"247c93102e37cb838fe7535ef0e685d264daf634816747cbec582a6eea2748ee"} Nov 03 20:08:50.961808 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:50.961786 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:51.017403 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.017363 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca\") pod \"6955eed5-7e8a-472f-a007-df4979408183\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " Nov 03 20:08:51.017571 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.017463 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca\") pod \"6955eed5-7e8a-472f-a007-df4979408183\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " Nov 03 20:08:51.017626 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.017610 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d\" (UniqueName: \"kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d\") pod \"6955eed5-7e8a-472f-a007-df4979408183\" (UID: \"6955eed5-7e8a-472f-a007-df4979408183\") " Nov 03 20:08:51.017834 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.017806 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca" pod "6955eed5-7e8a-472f-a007-df4979408183" (UID: "6955eed5-7e8a-472f-a007-df4979408183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:51.017958 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.017928 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6955eed5-7e8a-472f-a007-df4979408183-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:51.019450 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.019429 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca" pod "6955eed5-7e8a-472f-a007-df4979408183" (UID: "6955eed5-7e8a-472f-a007-df4979408183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:51.019653 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.019639 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d" pod "6955eed5-7e8a-472f-a007-df4979408183" (UID: "6955eed5-7e8a-472f-a007-df4979408183"). InnerVolumeSpecName "kube-api-access-5dc8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:51.118704 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.118626 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dc8d\" (UniqueName: \"kubernetes.io/projected/6955eed5-7e8a-472f-a007-df4979408183-kube-api-access-5dc8d\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:51.118704 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.118657 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6955eed5-7e8a-472f-a007-df4979408183-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:51.275885 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:08:51.275850 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:08:51.835300 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.835260 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" event={"ID":"6955eed5-7e8a-472f-a007-df4979408183","Type":"ContainerDied","Data":"922d77af655736f145b0a54f55c3e162a64247c45afb7bc52986b623390bc773"} Nov 03 20:08:51.835459 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.835317 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="922d77af655736f145b0a54f55c3e162a64247c45afb7bc52986b623390bc773" Nov 03 20:08:51.835459 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:51.835324 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mbdj" Nov 03 20:08:58.705317 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.705286 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx"] Nov 03 20:08:58.705898 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.705801 5233 cpu_manager.go:414] "containerMap: removing container" podUID="6955eed5-7e8a-472f-a007-df4979408183" containerName="revoke-keys" Nov 03 20:08:58.705898 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.705816 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6955eed5-7e8a-472f-a007-df4979408183" containerName="revoke-keys" Nov 03 20:08:58.706353 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.706328 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="6955eed5-7e8a-472f-a007-df4979408183" containerName="revoke-keys" Nov 03 20:08:58.712361 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.712325 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.717081 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.717031 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:58.721854 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.721809 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx"] Nov 03 20:08:58.722377 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.722350 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ktm9v\"" Nov 03 20:08:58.794587 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.794556 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ss2\" (UniqueName: \"kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.794761 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.794623 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.794832 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.794804 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.895533 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.895502 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.895743 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.895623 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.895743 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.895723 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ss2\" (UniqueName: \"kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.896391 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.896365 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.898126 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.898097 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:58.908415 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:58.908385 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ss2\" (UniqueName: \"kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2\") pod \"maas-api-revoke-sub-6b19338f-ftmxx\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:59.035787 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:59.035755 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:08:59.315833 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:59.315798 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx"] Nov 03 20:08:59.317557 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:08:59.317527 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925ddbec_d3d4_4b65_a2ac_ad31165b65d8.slice/crio-dcc8337e48f3aca6aa9e50295047b1e11114b5462204e219fe612490f84651e2 WatchSource:0}: Error finding container dcc8337e48f3aca6aa9e50295047b1e11114b5462204e219fe612490f84651e2: Status 404 returned error can't find the container with id dcc8337e48f3aca6aa9e50295047b1e11114b5462204e219fe612490f84651e2 Nov 03 20:08:59.879921 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:59.879837 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" event={"ID":"925ddbec-d3d4-4b65-a2ac-ad31165b65d8","Type":"ContainerStarted","Data":"de78b35aaf5d05c01533cda140ca7a25a7a3a5bb584b293aa0977505adbc0fd1"} Nov 03 20:08:59.879921 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:59.879893 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" event={"ID":"925ddbec-d3d4-4b65-a2ac-ad31165b65d8","Type":"ContainerStarted","Data":"dcc8337e48f3aca6aa9e50295047b1e11114b5462204e219fe612490f84651e2"} Nov 03 20:08:59.899964 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:08:59.899916 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" podStartSLOduration=1.899902024 podStartE2EDuration="1.899902024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:59.898511389 +0000 UTC m=+2506.239624891" watchObservedRunningTime="2026-11-03 20:08:59.899902024 +0000 UTC m=+2506.241015529" Nov 03 20:09:00.887284 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:00.887252 5233 generic.go:356] "Generic (PLEG): container finished" podID="925ddbec-d3d4-4b65-a2ac-ad31165b65d8" containerID="de78b35aaf5d05c01533cda140ca7a25a7a3a5bb584b293aa0977505adbc0fd1" exitCode=0 Nov 03 20:09:00.887683 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:00.887297 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" event={"ID":"925ddbec-d3d4-4b65-a2ac-ad31165b65d8","Type":"ContainerDied","Data":"de78b35aaf5d05c01533cda140ca7a25a7a3a5bb584b293aa0977505adbc0fd1"} Nov 03 20:09:02.031626 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.031601 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:09:02.127104 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.127071 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca\") pod \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " Nov 03 20:09:02.127298 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.127137 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca\") pod \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " Nov 03 20:09:02.127298 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.127291 5233 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2\" (UniqueName: \"kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2\") pod \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\" (UID: \"925ddbec-d3d4-4b65-a2ac-ad31165b65d8\") " Nov 03 20:09:02.127556 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.127526 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca" pod "925ddbec-d3d4-4b65-a2ac-ad31165b65d8" (UID: "925ddbec-d3d4-4b65-a2ac-ad31165b65d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:02.127665 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.127649 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-openshift-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:02.129657 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.129626 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca" pod "925ddbec-d3d4-4b65-a2ac-ad31165b65d8" (UID: "925ddbec-d3d4-4b65-a2ac-ad31165b65d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:02.129784 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.129765 5233 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2" pod "925ddbec-d3d4-4b65-a2ac-ad31165b65d8" (UID: "925ddbec-d3d4-4b65-a2ac-ad31165b65d8"). InnerVolumeSpecName "kube-api-access-g9ss2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:02.228426 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.228392 5233 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9ss2\" (UniqueName: \"kubernetes.io/projected/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-kube-api-access-g9ss2\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:02.228426 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.228426 5233 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925ddbec-d3d4-4b65-a2ac-ad31165b65d8-maas-api-xks-ca\") on node \"ip-10-0-128-162.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:02.900208 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.900181 5233 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" Nov 03 20:09:02.900389 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.900182 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ftmxx" event={"ID":"925ddbec-d3d4-4b65-a2ac-ad31165b65d8","Type":"ContainerDied","Data":"dcc8337e48f3aca6aa9e50295047b1e11114b5462204e219fe612490f84651e2"} Nov 03 20:09:02.900389 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:02.900299 5233 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc8337e48f3aca6aa9e50295047b1e11114b5462204e219fe612490f84651e2" Nov 03 20:09:10.051186 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:10.051151 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f46bb44fd-jzdlk_c2793993-763c-4b31-b090-69ac6ea8c04e/authorino/0.log" Nov 03 20:09:21.276621 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:09:21.276582 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:09:21.626459 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:21.626376 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjtbj_7c1dc26e-9699-469c-9432-6006a1210a69/manager/0.log" Nov 03 20:09:23.024002 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:23.023956 5233 scope.go:122] "RemoveContainer" containerID="4ca0fe0b449e6cab67105ca56939a28a0d743a83e8987f1f9570b1d54f742f74" Nov 03 20:09:23.930090 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:23.930060 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-4mbdj_6955eed5-7e8a-472f-a007-df4979408183/revoke-keys/0.log" Nov 03 20:09:24.113964 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:24.113936 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-2k2ts_27ee6f15-222b-41d7-92f1-1dba7a12a741/revoke-keys/0.log" Nov 03 20:09:24.304418 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:24.304390 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-m6292_826fbf06-281b-440c-b440-e1f714b28b3b/revoke-keys/0.log" Nov 03 20:09:24.476102 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:24.476072 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ftmxx_925ddbec-d3d4-4b65-a2ac-ad31165b65d8/revoke-keys/0.log" Nov 03 20:09:24.661015 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:24.660924 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8ef03f88-4bg78_f4a6091e-deb2-4bb3-84b7-5c9b82b39a49/revoke-keys/0.log" Nov 03 20:09:24.836774 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:24.836748 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dhhg5_0efceb7c-4394-442b-bc5b-72e0cba5d92c/revoke-keys/0.log" Nov 03 20:09:25.032676 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:25.032647 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-m8bbf_4a6e7a83-1671-4dc3-b972-a41676ac43a8/postgres/0.log" Nov 03 20:09:26.642726 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:26.642690 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f46bb44fd-jzdlk_c2793993-763c-4b31-b090-69ac6ea8c04e/authorino/0.log" Nov 03 20:09:27.095221 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:27.095189 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8p5ql_b11d7ca3-d9e1-4bed-8705-693f2351fe66/manager/0.log" Nov 03 20:09:27.488086 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:27.488052 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7dsgp_475ce0c0-4e75-4751-bdff-887b02e6c593/manager/0.log" Nov 03 20:09:27.918280 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:27.918197 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cdmlw_b2be9f43-240d-4ded-b132-5e15b27f689b/manager/0.log" Nov 03 20:09:28.750790 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:28.750764 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-8f76c657-2mnll_adaf19c0-adea-42f2-84f6-a889dd40eb81/istio-proxy/0.log" Nov 03 20:09:29.545020 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:29.544990 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/1.log" Nov 03 20:09:30.317154 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:30.317119 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-flqfr_a765c74d-cbf2-4d02-8212-a9635bb749ea/istio-proxy/0.log" Nov 03 20:09:30.519013 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:30.518984 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-fw2bg_0693dfcf-c351-424a-99b5-8dd89f1a2985/istio-proxy/0.log" Nov 03 20:09:30.751786 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:30.751759 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-jtkbc_dac123bf-79b0-466c-9546-18b94aadcbf1/istio-proxy/0.log" Nov 03 20:09:31.518368 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:31.518340 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf_f60ce991-dc22-432c-94f5-dd0f6b184c9c/istio-proxy/0.log" Nov 03 20:09:31.710608 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:31.710571 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2n225_784da5d0-b4e7-4590-a365-2be891af94c2/payload-pre-processing/0.log" Nov 03 20:09:31.900424 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:31.900353 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:09:32.576821 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:32.576791 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/0.log" Nov 03 20:09:33.321504 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:33.321477 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-852wk_cdb9e282-b5b5-457f-bbae-56f64d69ff9a/main/0.log" Nov 03 20:09:33.702343 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:33.702266 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qvdpc_92edadb3-9fae-48a3-aed1-54b25cf7c0ab/main/0.log" Nov 03 20:09:34.090338 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:34.090312 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mqbt7_f8189b43-c1cd-45ca-9f06-d2dd639e0fde/main/0.log" Nov 03 20:09:34.486335 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:34.486312 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4_91e4a136-8e90-43e8-86c5-0b666a31cd70/main/0.log" Nov 03 20:09:35.419801 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:35.419770 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f46bb44fd-jzdlk_c2793993-763c-4b31-b090-69ac6ea8c04e/authorino/0.log" Nov 03 20:09:45.421574 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:45.421546 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjtbj_7c1dc26e-9699-469c-9432-6006a1210a69/manager/0.log" Nov 03 20:09:47.534799 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:47.534768 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-4mbdj_6955eed5-7e8a-472f-a007-df4979408183/revoke-keys/0.log" Nov 03 20:09:47.738629 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:47.738603 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-2k2ts_27ee6f15-222b-41d7-92f1-1dba7a12a741/revoke-keys/0.log" Nov 03 20:09:47.954732 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:47.954651 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-m6292_826fbf06-281b-440c-b440-e1f714b28b3b/revoke-keys/0.log" Nov 03 20:09:48.173435 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:48.173407 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ftmxx_925ddbec-d3d4-4b65-a2ac-ad31165b65d8/revoke-keys/0.log" Nov 03 20:09:48.419278 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:48.419245 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8ef03f88-4bg78_f4a6091e-deb2-4bb3-84b7-5c9b82b39a49/revoke-keys/0.log" Nov 03 20:09:48.613463 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:48.613437 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-dhhg5_0efceb7c-4394-442b-bc5b-72e0cba5d92c/revoke-keys/0.log" Nov 03 20:09:48.796944 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:48.796919 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-m8bbf_4a6e7a83-1671-4dc3-b972-a41676ac43a8/postgres/0.log" Nov 03 20:09:49.970345 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:49.970317 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f46bb44fd-jzdlk_c2793993-763c-4b31-b090-69ac6ea8c04e/authorino/0.log" Nov 03 20:09:50.382853 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:50.382771 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8p5ql_b11d7ca3-d9e1-4bed-8705-693f2351fe66/manager/0.log" Nov 03 20:09:50.770984 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:50.770932 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7dsgp_475ce0c0-4e75-4751-bdff-887b02e6c593/manager/0.log" Nov 03 20:09:51.185141 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:51.185064 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cdmlw_b2be9f43-240d-4ded-b132-5e15b27f689b/manager/0.log" Nov 03 20:09:51.277753 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:09:51.277720 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:09:51.790901 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:51.790866 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-8f76c657-2mnll_adaf19c0-adea-42f2-84f6-a889dd40eb81/istio-proxy/0.log" Nov 03 20:09:52.560783 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:52.560751 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/1.log" Nov 03 20:09:53.314552 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:53.314523 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-flqfr_a765c74d-cbf2-4d02-8212-a9635bb749ea/istio-proxy/0.log" Nov 03 20:09:53.493768 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:53.493689 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-fw2bg_0693dfcf-c351-424a-99b5-8dd89f1a2985/istio-proxy/0.log" Nov 03 20:09:53.685354 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:53.685275 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-jtkbc_dac123bf-79b0-466c-9546-18b94aadcbf1/istio-proxy/0.log" Nov 03 20:09:54.029193 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:54.029158 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292"] Nov 03 20:09:54.031744 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:54.031714 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6292"] Nov 03 20:09:54.441711 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:54.441629 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf_f60ce991-dc22-432c-94f5-dd0f6b184c9c/istio-proxy/0.log" Nov 03 20:09:54.629147 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:54.629119 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2n225_784da5d0-b4e7-4590-a365-2be891af94c2/payload-pre-processing/0.log" Nov 03 20:09:54.827457 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:54.827427 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:09:55.762919 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:55.762889 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-852wk_cdb9e282-b5b5-457f-bbae-56f64d69ff9a/main/0.log" Nov 03 20:09:55.835585 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:55.835556 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826fbf06-281b-440c-b440-e1f714b28b3b" path="/var/lib/kubelet/pods/826fbf06-281b-440c-b440-e1f714b28b3b/volumes" Nov 03 20:09:56.120377 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:56.120306 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qvdpc_92edadb3-9fae-48a3-aed1-54b25cf7c0ab/main/0.log" Nov 03 20:09:56.485782 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:56.485754 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mqbt7_f8189b43-c1cd-45ca-9f06-d2dd639e0fde/main/0.log" Nov 03 20:09:56.871868 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:09:56.871796 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-26bc4_91e4a136-8e90-43e8-86c5-0b666a31cd70/main/0.log" Nov 03 20:10:08.025331 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:08.025296 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts"] Nov 03 20:10:08.027608 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:08.027584 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2k2ts"] Nov 03 20:10:09.836678 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:09.836644 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ee6f15-222b-41d7-92f1-1dba7a12a741" path="/var/lib/kubelet/pods/27ee6f15-222b-41d7-92f1-1dba7a12a741/volumes" Nov 03 20:10:21.276757 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:10:21.276722 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:10:23.046825 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:23.046792 5233 scope.go:122] "RemoveContainer" containerID="ec5182379fed0022e06a39cadbebfa310cb15ec8d0b7dd7c46b20f55d079f5f4" Nov 03 20:10:23.058553 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:23.058515 5233 scope.go:122] "RemoveContainer" containerID="b59a4fd2ea60defa8147e4925a7c61d27af9f412b7af400ec5fd4ab1a67068eb" Nov 03 20:10:27.033433 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:27.033393 5233 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78"] Nov 03 20:10:27.037595 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:27.037566 5233 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ef03f88-4bg78"] Nov 03 20:10:27.835806 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:27.835767 5233 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a6091e-deb2-4bb3-84b7-5c9b82b39a49" path="/var/lib/kubelet/pods/f4a6091e-deb2-4bb3-84b7-5c9b82b39a49/volumes" Nov 03 20:10:35.966772 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:35.966739 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:10:36.146852 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:10:36.146819 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:10:51.277005 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:10:51.276948 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:11:21.276267 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:11:21.276230 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:11:23.084361 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:23.084329 5233 scope.go:122] "RemoveContainer" containerID="0098ee419f9003187fa1cd00db1b55462013c1bf05f4642bee7925d52d1ad798" Nov 03 20:11:23.093153 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:23.093127 5233 scope.go:122] "RemoveContainer" containerID="0e1730f72590914d8c07de528eb6c0a9bd43af957b859980beac74490b03b7ef" Nov 03 20:11:23.118516 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:23.118488 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:11:23.296610 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:23.296585 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:11:26.808883 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:26.808845 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s9hlp_9c908f51-a28e-46fb-af36-ac4cb9163671/global-pull-secret-syncer/0.log" Nov 03 20:11:27.168733 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:27.168654 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f5dt8_4936d277-165f-4cf6-a9cb-6930c76e875a/konnectivity-agent/0.log" Nov 03 20:11:27.198603 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:27.198575 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-162.us-east-2.compute.internal_9c8881dae007849f880e1281962f0bcb/haproxy/0.log" Nov 03 20:11:33.401588 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:33.401551 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f46bb44fd-jzdlk_c2793993-763c-4b31-b090-69ac6ea8c04e/authorino/0.log" Nov 03 20:11:33.464982 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:33.464956 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8p5ql_b11d7ca3-d9e1-4bed-8705-693f2351fe66/manager/0.log" Nov 03 20:11:33.560560 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:33.560528 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7dsgp_475ce0c0-4e75-4751-bdff-887b02e6c593/manager/0.log" Nov 03 20:11:33.651319 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:33.651290 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cdmlw_b2be9f43-240d-4ded-b132-5e15b27f689b/manager/0.log" Nov 03 20:11:35.160532 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.160504 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z69vm_ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/kube-state-metrics/0.log" Nov 03 20:11:35.195147 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.195114 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z69vm_ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/kube-rbac-proxy-main/0.log" Nov 03 20:11:35.229410 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.229385 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z69vm_ea3dcb7c-3e19-41bb-ba36-8c2afbde6bd5/kube-rbac-proxy-self/0.log" Nov 03 20:11:35.294743 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.294715 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-qf99w_4ea05d8b-7c65-43b2-b74e-71b25b7f9da2/monitoring-plugin/0.log" Nov 03 20:11:35.335933 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.335904 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdzp5_c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/node-exporter/0.log" Nov 03 20:11:35.361481 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.361455 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdzp5_c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/kube-rbac-proxy/0.log" Nov 03 20:11:35.392983 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.392949 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdzp5_c87c6d25-883d-4f0d-b4d3-8c77abbb41ea/init-textfile/0.log" Nov 03 20:11:35.619780 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.619727 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zc8b5_0133e827-7681-4c7a-a5a8-34b9c5da874c/kube-rbac-proxy-main/0.log" Nov 03 20:11:35.648797 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.648777 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zc8b5_0133e827-7681-4c7a-a5a8-34b9c5da874c/kube-rbac-proxy-self/0.log" Nov 03 20:11:35.678932 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.678905 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zc8b5_0133e827-7681-4c7a-a5a8-34b9c5da874c/openshift-state-metrics/0.log" Nov 03 20:11:35.723178 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.723152 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/prometheus/0.log" Nov 03 20:11:35.771534 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.771512 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/config-reloader/0.log" Nov 03 20:11:35.801456 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.801437 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/thanos-sidecar/0.log" Nov 03 20:11:35.828884 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.828863 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/kube-rbac-proxy-web/0.log" Nov 03 20:11:35.857144 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.857123 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/kube-rbac-proxy/0.log" Nov 03 20:11:35.884409 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.884337 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/kube-rbac-proxy-thanos/0.log" Nov 03 20:11:35.914730 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:35.914709 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb066071-701f-4f31-84b0-f1e4bbb906eb/init-config-reloader/0.log" Nov 03 20:11:36.141180 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.141084 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f68bfd554-gj6f6_a92b0d62-15bf-4769-8a0c-fa53b1079ca2/thanos-query/0.log" Nov 03 20:11:36.167966 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.167944 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f68bfd554-gj6f6_a92b0d62-15bf-4769-8a0c-fa53b1079ca2/kube-rbac-proxy-web/0.log" Nov 03 20:11:36.195197 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.195171 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f68bfd554-gj6f6_a92b0d62-15bf-4769-8a0c-fa53b1079ca2/kube-rbac-proxy/0.log" Nov 03 20:11:36.226541 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.226516 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f68bfd554-gj6f6_a92b0d62-15bf-4769-8a0c-fa53b1079ca2/prom-label-proxy/0.log" Nov 03 20:11:36.253888 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.253862 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f68bfd554-gj6f6_a92b0d62-15bf-4769-8a0c-fa53b1079ca2/kube-rbac-proxy-rules/0.log" Nov 03 20:11:36.282237 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.282215 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f68bfd554-gj6f6_a92b0d62-15bf-4769-8a0c-fa53b1079ca2/kube-rbac-proxy-metrics/0.log" Nov 03 20:11:36.777193 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.777163 5233 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c"] Nov 03 20:11:36.777611 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.777599 5233 cpu_manager.go:414] "containerMap: removing container" podUID="925ddbec-d3d4-4b65-a2ac-ad31165b65d8" containerName="revoke-keys" Nov 03 20:11:36.777675 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.777613 5233 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925ddbec-d3d4-4b65-a2ac-ad31165b65d8" containerName="revoke-keys" Nov 03 20:11:36.777716 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.777703 5233 memory_manager.go:365] "RemoveStaleState removing state" podUID="925ddbec-d3d4-4b65-a2ac-ad31165b65d8" containerName="revoke-keys" Nov 03 20:11:36.782653 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.782635 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:36.788525 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.788500 5233 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b8lpv\"/\"default-dockercfg-tjpxw\"" Nov 03 20:11:36.788525 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.788515 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b8lpv\"/\"kube-root-ca.crt\"" Nov 03 20:11:36.788677 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.788540 5233 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b8lpv\"/\"openshift-service-ca.crt\"" Nov 03 20:11:36.788765 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.788747 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c"] Nov 03 20:11:36.899601 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.899569 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-lib-modules\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:36.899842 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.899819 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-proc\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:36.899901 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.899871 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxd4\" (UniqueName: \"kubernetes.io/projected/e1ec9e2e-e380-49ac-a936-70a145f4f75f-kube-api-access-srxd4\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:36.899954 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.899940 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-podres\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:36.900019 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:36.899980 5233 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-sys\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000680 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000648 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-lib-modules\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000819 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000697 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-proc\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000819 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000722 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxd4\" (UniqueName: \"kubernetes.io/projected/e1ec9e2e-e380-49ac-a936-70a145f4f75f-kube-api-access-srxd4\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000819 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000753 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-podres\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000819 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000769 5233 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-sys\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000819 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000775 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-proc\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.000819 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000783 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-lib-modules\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.001076 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000848 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-sys\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.001076 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.000882 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1ec9e2e-e380-49ac-a936-70a145f4f75f-podres\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.013239 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.013210 5233 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxd4\" (UniqueName: \"kubernetes.io/projected/e1ec9e2e-e380-49ac-a936-70a145f4f75f-kube-api-access-srxd4\") pod \"perf-node-gather-daemonset-grf6c\" (UID: \"e1ec9e2e-e380-49ac-a936-70a145f4f75f\") " pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.099923 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.099859 5233 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.379324 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.379298 5233 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c"] Nov 03 20:11:37.381932 ip-10-0-128-162 kubenswrapper[5233]: W1103 20:11:37.381905 5233 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1ec9e2e_e380_49ac_a936_70a145f4f75f.slice/crio-26a9ece2274763f8c91ca2605b4178da238563c41b6bed507cd7af4d39dcbd5c WatchSource:0}: Error finding container 26a9ece2274763f8c91ca2605b4178da238563c41b6bed507cd7af4d39dcbd5c: Status 404 returned error can't find the container with id 26a9ece2274763f8c91ca2605b4178da238563c41b6bed507cd7af4d39dcbd5c Nov 03 20:11:37.918086 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.918049 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" event={"ID":"e1ec9e2e-e380-49ac-a936-70a145f4f75f","Type":"ContainerStarted","Data":"459e170ae6411f545462dd08ba57c9cfe7664a7ee36aebe56123e6ef865c7a40"} Nov 03 20:11:37.918263 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.918110 5233 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" event={"ID":"e1ec9e2e-e380-49ac-a936-70a145f4f75f","Type":"ContainerStarted","Data":"26a9ece2274763f8c91ca2605b4178da238563c41b6bed507cd7af4d39dcbd5c"} Nov 03 20:11:37.918263 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.918231 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:37.941121 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:37.941078 5233 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" podStartSLOduration=1.94106738 podStartE2EDuration="1.94106738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:37.938211642 +0000 UTC m=+2664.279325127" watchObservedRunningTime="2026-11-03 20:11:37.94106738 +0000 UTC m=+2664.282180878" Nov 03 20:11:40.627727 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:40.627699 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7fzhd_d1f99516-4d86-4bdd-8df3-39c1ec90f28a/dns/0.log" Nov 03 20:11:40.654057 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:40.654032 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7fzhd_d1f99516-4d86-4bdd-8df3-39c1ec90f28a/kube-rbac-proxy/0.log" Nov 03 20:11:40.804161 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:40.804133 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dmtq4_89460d5b-9bf2-4029-9d64-1e21733554d0/dns-node-resolver/0.log" Nov 03 20:11:41.570177 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:41.570148 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22xws_21d46003-1ed7-40dd-b4b8-1e3a05448eee/node-ca/0.log" Nov 03 20:11:42.672659 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:42.672630 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-8f76c657-2mnll_adaf19c0-adea-42f2-84f6-a889dd40eb81/istio-proxy/0.log" Nov 03 20:11:43.934172 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:43.934142 5233 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b8lpv/perf-node-gather-daemonset-grf6c" Nov 03 20:11:45.380352 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.380323 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/0.log" Nov 03 20:11:45.381136 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.381119 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/1.log" Nov 03 20:11:45.518899 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.518867 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-flqfr_a765c74d-cbf2-4d02-8212-a9635bb749ea/istio-proxy/0.log" Nov 03 20:11:45.556228 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.556204 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-fw2bg_0693dfcf-c351-424a-99b5-8dd89f1a2985/istio-proxy/0.log" Nov 03 20:11:45.590706 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.590685 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-jtkbc_dac123bf-79b0-466c-9546-18b94aadcbf1/istio-proxy/0.log" Nov 03 20:11:45.789515 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.789483 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf_f60ce991-dc22-432c-94f5-dd0f6b184c9c/istio-proxy/0.log" Nov 03 20:11:45.847376 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.847351 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2n225_784da5d0-b4e7-4590-a365-2be891af94c2/payload-pre-processing/0.log" Nov 03 20:11:45.887079 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:45.887055 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:11:47.034448 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:47.034419 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-8f76c657-2mnll_adaf19c0-adea-42f2-84f6-a889dd40eb81/istio-proxy/0.log" Nov 03 20:11:47.892823 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:47.892794 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xtdsf_652e8df9-bec9-4bdc-9535-4f2ea47f45d6/serve-healthcheck-canary/0.log" Nov 03 20:11:48.684360 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:48.684331 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-npmtd_1e9c8905-cc87-498c-8a3a-ec294b27fb9d/kube-rbac-proxy/0.log" Nov 03 20:11:48.717229 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:48.717192 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-npmtd_1e9c8905-cc87-498c-8a3a-ec294b27fb9d/exporter/0.log" Nov 03 20:11:48.748568 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:48.748543 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-npmtd_1e9c8905-cc87-498c-8a3a-ec294b27fb9d/extractor/0.log" Nov 03 20:11:49.289340 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.289312 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/0.log" Nov 03 20:11:49.290566 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.290546 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-8tfdw_d6929ca2-35e3-40c0-8010-49b9dc0a68c5/istio-proxy/1.log" Nov 03 20:11:49.371683 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.371657 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-flqfr_a765c74d-cbf2-4d02-8212-a9635bb749ea/istio-proxy/0.log" Nov 03 20:11:49.391872 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.391850 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-fw2bg_0693dfcf-c351-424a-99b5-8dd89f1a2985/istio-proxy/0.log" Nov 03 20:11:49.410979 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.410953 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-jtkbc_dac123bf-79b0-466c-9546-18b94aadcbf1/istio-proxy/0.log" Nov 03 20:11:49.489566 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.489541 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6d5bbf59c5-rptjf_f60ce991-dc22-432c-94f5-dd0f6b184c9c/istio-proxy/0.log" Nov 03 20:11:49.506191 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.506163 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2n225_784da5d0-b4e7-4590-a365-2be891af94c2/payload-pre-processing/0.log" Nov 03 20:11:49.523873 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:49.523852 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-57shs_bea15765-cb00-498d-a197-4550768d770b/payload-processing/0.log" Nov 03 20:11:51.203735 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:51.203704 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7646b49899-f98ww_1eb6b218-6ded-42a8-9667-3bec01071e87/manager/0.log" Nov 03 20:11:51.230452 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:51.230429 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7646b49899-f98ww_1eb6b218-6ded-42a8-9667-3bec01071e87/copy-manifests/0.log" Nov 03 20:11:51.276417 ip-10-0-128-162 kubenswrapper[5233]: E1103 20:11:51.276386 5233 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-npmtd" containerName="extractor" Nov 03 20:11:51.888150 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:51.888118 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-8f76c657-2mnll_adaf19c0-adea-42f2-84f6-a889dd40eb81/istio-proxy/0.log" Nov 03 20:11:52.568962 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:52.568925 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-864947c86-tbcbq_1d798784-e901-4515-9385-47efdb181643/manager/0.log" Nov 03 20:11:53.482296 ip-10-0-128-162 kubenswrapper[5233]: I1103 20:11:53.482253 5233 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mjtbj_7c1dc26e-9699-469c-9432-6006a1210a69/manager/0.log"